Trace Analyzer

The Trace Analyzer provides a real-time view of incoming traces grouped by tags and their relative frequency. Use this information to help understand how often your applications emit traces, troubleshoot spikes in ingest rates, and ensure that the Collector is aware of particular kinds of traces.

Trace Explorer operates on persisted data only, whereas Trace Analyzer can show data at any stage in the ingest pipeline, such as ingested, dropped, and persisted data. If you configured tail sampling, which occurs after data ingestion, you can use Trace Analyzer to see spans that were dropped based on tail sampling rules.

Prerequisites

To explore tracing data in Chronosphere, you must either install and configure the Chronosphere-provided Collector or the OpenTelemetry Collector to receive trace data from your services.

Profile incoming traces

You must have administrative privileges to complete this task.

To profile incoming traces:

1. Click Go to Admin.

2. In the navigation menu, select Analyzers > Live Telemetry Analyzer, and then click the Traces tab.

3. Click Live to start profiling traces.

4. To group incoming traces by tags, enter the tag value in the Group by tags field.

5. To filter traces by tag across all incoming traces, enter a the tag and value to filter on as a key:value pair in the Filter by tags field.

   To filter results to only root spans, enter __root_svc__:true. Root spans are the top-level spans that all other spans are children or ancestors of. You can safely sample these spans without impacting the integrity of a trace.

You can use glob syntax in this field to filter on specific patterns.

View incoming traces

Trace Analyzer displays data in one table for tags and one table for spans.

Tags table

The leftmost table displays available tag values, including the tag name, how often it appears, its unique values, and how much space it consumes. This table includes these columns:

• Appears In: The percentage of spans you're viewing that have the matching label key. Use this column to locate tags to dissect traces. A trace with a high total percentage and low unique values indicates a high-level breakdown of where to attribute traces to.
• Unique: Number of unique values for the respective label key. Use this column to find high cardinality tags.
• Total size (bytes): The total size, in bytes, of each tag from when you click Live.
• Avg size (bytes): The average size, in bytes, of each tag.

For example, consider the following spans:

span1: __svc__ = "sign_up", location = "loc1", version = "0.100"
span2: __svc__ = "sign_up", location = "loc2", version = "0.98"
span3: __svc__ = "login", version = "0.98"

These traces generate three rows in the leftmost table because there are three tags: __svc__, location, and version. Every trace has an __svc__ tag, so the total percentage is always 100%. Only two traces have the location tag, so the total percentage is 66%, which is also true for the version tag.

There are only two unique values for __svc__, sign_up, and login, so the Unique column displays 2. Only two traces have the location tag, which has two unique values: loc1 and loc2.

Spans table

The rightmost table displays the average and current spans per second for the selected tags in the Avg. SPPS and Current SPPS columns. The table groups data by the selected tag values, which display in the first columns. For example, if you enter __svc__ in the Group by Tags field, that tag displays as the first column in this table. Like in the tags table, the spans table includes total and average span sizes for every tag value.

• Avg SPPS: Average rate of spans processed in spans per second.
• Current SPPS: Current rate of spans processed in spans per second.
• Total size (bytes): The total size, in bytes, of each span from when you click Live.
• Avg size (bytes): The average size, in bytes, of each span associated with the label specified in the Group by Tags field.

Group and filter spans

You can filter or group traces by tag using the Group by tags and Filter by tags fields. If you select multiple tags, the table lists the groups in the order you select them.

Trace Analyzer groups traces using the __svc__ tag by default. Select a tag in the Group by tag field to add it as a column in the rightmost table. To filter on a specific tag, enter a tag and value in the Filter by tag field as a key:value pair. For example, you can enter service.name:traceingester to filter specifically on that key:value pair.

Use the Showing list to filter incoming traces based on their status:

• Ingested: Traces that are still processing for storage.
• Dropped: Traces that dropped based on defined drop rules.
• Missing Root: Traces that are missing their root span.