Alauda Service Mesh

Incomplete Trace Data

TOC

Problem DescriptionRoot Cause Analysis1. Data Persistence Delay2. Time Range LimitationSolution for Root Cause 1Solution for Root Cause 2

Problem Description

The trace query results exhibit the following issues of incomplete data:

  • Recent queries (within the last 30 minutes) are missing some spans.
  • Trace data older than 1 hour are experiencing disconnections.

Root Cause Analysis

1. Data Persistence Delay

The writing process in Elasticsearch requires a sequence of steps involving memory buffer → translog → segment files, which can result in visibility delays for the most recently written data.

2. Time Range Limitation

By default, when jaeger-query queries spans corresponding to traces, the time range extends one hour before and after the start time of the span.

For instance, if a span starts at 08:12:30 and ends at 08:12:32, the time range for querying that trace would be from 07:12:30 to 09:12:32.

Therefore, if the trace spans over 1 hour, querying through this span may not yield a complete trace.

Solution for Root Cause 1

Wait a moment and refresh the page to try the query again.

Solution for Root Cause 2

If the trace span in your environment is lengthy, you can adjust the query time range for a single trace using the --es.asm.span-trace-query-time-adjustment-hours startup parameter in jaeger-query.

The default value of this parameter is 1 hour, and you can increase this value as needed.