Skip to main content
Trace Analyzer provides a real-time view of incoming spans as Chronosphere Observability Platform processes them. Use Trace Analyzer to compare spans by tag, troubleshoot spikes in ingest rates, and inspect spans before they reach storage. Trace Explorer queries persisted traces, whereas Trace Analyzer profiles spans in the trace ingester. Trace Analyzer classifies spans as ingested, dropped, or part of a trace that’s missing its root span. Dropped spans include spans discarded by tail sampling, behavior-based sampling, or product limits.

​
Prerequisites

To explore tracing data in Observability Platform, you must either install and configure the Chronosphere Collector or the OpenTelemetry Collector to receive trace data from your services.

​
Profile incoming traces

To profile incoming traces:
  1. In the navigation menu, click Go to Admin and then select Analyzers > Live Telemetry.
  2. Click the Traces tab.
  3. Click Capture live data to start profiling traces. Trace Analyzer automatically pauses capture after three minutes. Click Capture live data to resume capture.
  4. To group incoming spans by tags, select up to 10 tag keys in the Group by tags field.
  5. To filter spans by tag, enter a tag key and value as a key:value pair in the Filter by tags field. To display only root spans, enter __root_service__:true or __root_operation__:true. A root span has no parent and represents the top-level operation in a trace.
Use glob syntax in the Filter by tags field to match specific patterns. Trace Analyzer applies all filters together.

​
View incoming traces

Trace Analyzer displays a Tags table and a Tag values table. By default, each table displays up to 100 rows and calculates statistics from a rolling 15-second window. Select a numeric column heading to sort its values.

​
Tags table

The Tags table displays available tag keys, how often each key appears, an estimate of its unique values, and the size of its data. Click a tag key to add it to the Group by tags field. The table contains these columns:
  • Tag key: The tag key.
  • Appears in: The percentage of spans in the rolling window that contain the tag key. Use this column to find tags that provide a broad breakdown of spans.
  • Unique value: The estimated number of unique values for the tag key. Use this column to identify tags with many distinct values.
  • Total size (bytes): The total size of the tag data in the rolling window.
  • Avg. size (bytes): The average size of the tag data.
For example, consider the following spans:
These spans generate three rows because the example contains three tag keys: __service__, location, and version. All three spans contain the __service__ and version tag keys, so Appears in displays 100% for those rows. Two of the three spans contain location, which displays as 67% after rounding. Each tag key has two unique values, so Unique value displays 2 for every row. The following table shows the statistics generated by the example spans:

​
Tag values table

The Tag values table groups spans by the tag keys selected in the Group by tags field. The selected tag keys appear as the first columns and in the order selected. Click a tag value to add it to the Filter by tags field. The table contains these additional columns:
  • Avg. spans/sec: The average rate of spans processed each second during the rolling window.
  • Total size (bytes): The total size of the grouped spans in the rolling window.
  • Avg. size (bytes): The average size of the grouped spans.
The following table shows example statistics when grouping by __service__:

​
Promote tags for differential diagnosis

When viewing a stream of incoming traces, the tags table displays all tags related to your trace data. If you pause the live stream of traces, you can select individual tags and promote them to top tags in the Differential Diagnosis tab of Trace Explorer. Promoting tags makes them display in all differential diagnosis results for every user in your Observability Platform tenant. You must have administrative privileges to complete this task. To promote tags to top tags:
  1. In the navigation menu, click Go to Admin and then select Analyzers > Live Telemetry.
  2. Click the Traces tab.
  3. Click Capture live data to start profiling traces.
  4. Click Capture live data again to pause the live stream of traces.
  5. Hold the pointer over the tag you want to promote, click the three vertical dots icon, and then click Add to top tags. If the tag has more than 250 unique values, a confirmation dialog appears. Click Add to confirm. Observability Platform adds the tag to the tenant’s top tags. Trace Explorer then includes the tag in results on the Differential Diagnosis tab. See Access and use differential diagnosis for more information.
  6. Complete the previous step to promote additional tags.

​
Group and filter spans

Trace Analyzer groups spans by __service__ and displays ingested spans by default. Use the controls in these ways:
  • In Group by tags, select up to 10 tag keys. The Tag values table adds a column for each key in the order selected. Clicking a key in the Tags table also adds it to this field.
  • In Filter by tags, enter key:value pairs. For example, enter service.name:traceingester. Clicking a value in the Tag values table also adds it to this field.
  • In Data phase, select one or more span classifications:
    • Ingested: Spans observed as they arrive during trace processing.
    • Dropped: Spans discarded by sampling decisions or product limits.
    • Missing root: Spans from traces that don’t contain a root span.