Identify issues behind suspicious trends

When an incident occurs, you can use Trace Explorer to search and filter trace data to identify trends in requests, errors, or latencies in services and related operations. After identifying a trend, you want to understand why it's happening.

Differential diagnosis lets you identify trends and immediately scan through all related tags and values to pinpoint the exact tag:value pairs most closely correlated with suspicious behavior.

Administrators can promote top tags in your Observability Platform tenant so those tags always display in differential diagnosis results.

Access and use differential diagnosis

You can access differential diagnosis insights from different areas of Trace Explorer. The method you choose depends on your entry point and what data you're investigating:

• Statistics: Use this tab to locate interesting or alarming trends in errors or latencies. After identifying a service or service and operation with increasing trends, use differential diagnosis to understand what issues are causing the trend.

• Differential Diagnosis: Use this tab to check the health of the service or service and operation your team owns. Go directly to this tab to determine whether errors, successes, and latencies are distributed across environments, regions, build versions, or other important contexts.

• Traces: Use this tab to explore a specific trace that you received an alert about, or discovered in a related log. Locate spans with very long durations or that are generating considerable errors, and then use differential diagnosis to complete an aggregate analysis on a larger set of similar spans to elicit a common trend that might explain unexpected or irregular behavior.

To access and use differential diagnosis:

1. In the navigation menu select Explorers > Trace Explorer.

2. Choose the time period to display results for, which defaults to the last 15 minutes.

   You must choose a time window of one hour or less to display differential diagnosis insights.

3. Use the Query builder to define a search, and then click Run.

4. Use one of the following options to access differential diagnosis insights:

   • On the Statistics tab, click a service or service and operation, and then click Differential Diagnosis in the resulting dialog.

   • Click the Differential Diagnosis tab. Select a service or service and operation.

     A shortcut to view traces related to your differential diagnosis query is to click View traces on the Differential Diagnosis tab. The resulting dialog displays individual traces matching your query in the same view presented in the Traces tab. Click Update Trace Explorer query to apply the differential diagnosis criteria you specified to the Query builder in Trace Explorer.

   • Click the Traces tab. Click the name of a trace in the Trace column to open the trace details page. Select a span, and then click Differential Diagnosis.

     The Differential Diagnosis tab displays insights for the criteria you selected.

5. Select one or more tags to display the distribution of those tag:value pairs across several metrics simultaneously.

6. To sync the order of all the bars in other charts to a specific chart, use the Chart sorting dropdown. For example, if you choose Sync to error spans, each of the charts update to reflect the ordering of the Error spans chart. This capability lets you compare the same tag across different heuristics.

Promote top tags

Administrators can promote tags to display in the Top tags section of the Differential Diagnosis tab. These tags always display in differential diagnosis results for all users in your Observability Platform tenant.

You must have administrative privileges to complete this task.

To promote tags to top tags:

1. In the navigation menu click Go to Admin and then select Explorers > Trace Explorer.

2. Click the Differential Diagnosis tab.

3. In the Find tags field, enter the name of the tag you want to promote to a top tag.

4. Hold the pointer over the tag, click the three vertical dots icon, and then click Add to top tags.

   The selected tag moves to the Top tags section, and is automatically included in differential diagnosis results.

5. Complete the previous step to promote additional tags.

To remove a tag from the Top tags section, hold the pointer over the tag, click the three vertical dots icon, and then click Remove from top tags.