Identify issues behind suspicious metrics trends

Differential diagnosis (DDx) for metrics is a part of the metric exploration and hypothesis testing workflow in Chronosphere Observability Platform. This feature helps users who encounter time series anomalies, such as unusual shapes in their metrics like spikes and dips, quickly understand what dimensions are contributing to an issue. Knowing what dimensions contribute to an issue can help you correlate patterns.

Differential diagnosis supports Prometheus metrics.

Differential diagnosis provides the following benefits:

• Dimension exploration: Given an arbitrary query with an anomaly, users can quickly see all the dimensions available for the metrics present. They can use this information to understand the surface area of the available for exploration. Dimensions can be included and excluded from the results to enable hypothesis testing.
• Automatic data analysis: Observability Platform automatically explores data using variations of the user’s query, leveraging all available dimensions. Observability Platform compares the results to the original query and surfaces the series that are most closely correlated with the anomaly.

Differential diagnosis uses correlated data to help you identify the metrics contributing to a problem. Correlated data is data that contributes the most to a graph. For example, if Prod is highly correlated, you might want to use a filter to isolate metrics from that environment. After identifying the next highly correlated item, use additional filters to focus on the problem.

Access differential diagnosis

You can access differential diagnosis insights from different areas of Metrics Explorer.

1. In the navigation menu select Explorers > Metrics Explorer.
2. In the query section, click DDx. The Metrics DDx page displays.
3. Create a query to define a search.
4. Click Run.

If you create a query in Metrics Explorer, access differential diagnosis from any graphs generated by that query.

1. Generate a query in Metrics Explorer.
2. In any graph, click the three vertical dots and then select Metrics DDx. The Metrics DDx page opens, with the query box pre-populated by the query used in the explorer.

The query runs and results display in the page.

In any Metrics DDx graph, click the three vertical dots access additional options:

• Open in Metrics Explorer: Open Metrics Explorer using the query generated by this graph.
• Add to dashboard: Add this graph to a dashboard.

Filter data

You can use pinned scopes to filter your data. When using a pinned scope, the scope displays in the Pinned scopes box, and again after the query.

Use pins in label graphs to add a filter:

1. Hold the pointer over a series in the graph and click it to pin the series.
2. In the dialog box, click to add that label value to the filters.

To remove the filter, click the x on the filter’s chip or click in the pinned box. Click Reset filters to remove all filters except pinned scopes. Pinned scopes can only be removed by deleting them from the Pinned scopes box.

X-ray

Observability Platform uses derived metrics to reduce query time and complexity. The X-ray feature lets you explore the queries using derived metrics so you can see which specific metric correlates with the problem you’re seeing.

Understand the Metrics DDx graphs

The Metrics DDx page displays information related to your query. Use the time picker to change the time range of the query results to modify the displayed time period.

The query generates a graph matching the query in Metrics Explorer, and a Correlation comparison graph. The Correlation comparison graph displays the query aggregated to a single metric used to find correlations. Observability Platform uses sum by default. If you use another aggregation in your query, Observability Platform uses that aggregation over the entire original query. For example, if you use max, then Observability Platform uses that aggregation over the entire query.

The Correlation summary is a list of labels and values that the query for the Correlation comparison graph uses. These include labels like:

• grpc_service
• grpc_method
• environment

Labels breakdown

Use the Labels breakdown section to search for a label. The number next to a label indicates the number of label values. Sort labels by Correlation or Name. Labels with high correlation values contribute more data to a given graph.

Observability Platform indicates high correlation values with a bar chart next to the label value. Higher correlated values display more bars, to a maximum of three bars. Hold the pointer over the bar chart to display the Correlation score.

For each label in the original query, Observability Platform groups by the label and renders a correlation graph with all the label values. For example, the container label might have two values: recommendationservice and <empty>, which Observability Platform shows as <no value> meaning some metric series don’t have this label.

Observability Platform shows a graph for each label name by default. Switch the One graph per dropdown to drill down and render a graph per value for any single label. Single cardinality labels are hidden because they’re the same shape as the comparison graph. Click the arrow to expand this section.

Toggle Show comparison to overlay the Correlation comparison graph results on the cardinality label graphs. Labels with a single cardinality are hidden because they’re the same shape as the comparison graph.