Search and filter trace data

In Trace Explorer, use the Query builder to search traces by their unique ID, or by the details of the individual spans in a given trace. You can add one or more span filters to additionally refine the trace results. These filters match spans within a trace based on criteria you select. You can choose to include or exclude operations, services, or tags from your search:

• An include statement narrows trace results to only include those that contain at least one span that matches the query.
• An exclude statement narrows trace results to only those with no spans that match the span query.

Including multiple span criteria in a single row constitutes an AND evaluation. You can define a query in a single row or on multiple rows, which equate to different evaluations:

• Single query row: If you enter multiple criteria in the same query row, Chronosphere Observability Platform evaluates them as AND criteria for any single span in a trace. For example, the following query returns spans that include both the ordering-svc service and the /ordering.Ordering/FetchItem operation:

  service="ordering-svc" operation="/ordering.Ordering/FetchItem"

• Multiple query rows: If you specify queries on multiple rows, Observability Platform evaluates all criteria in the first query against all possible matching spans, and then evaluates criteria in subsequent queries against matching spans.

  Observability Platform only returns traces where at least one span matches the first query and also at least one more span matches additional queries.

  Using multiple query rows equates to, "Find all traces with at least one span matching the first span criteria, and also at least one span matching this other criteria." Each query row is equal to another span in the trace.

  The previous query, when defined on separate query rows, returns spans that include the ordering-svc service, and at least one other span that includes the /ordering.Ordering/FetchItem operation:

  service="ordering-svc"
  
  operation="/ordering.Ordering/FetchItem"

Define a search

You might want to search for any traces that include a span with an operation named /paymentstore.PaymentStore/Capture, which captures all payment information to your app. You can then add another span filter in a new row to exclude traces where any other span contains a tag like geo=fr to filter out any payments made in France.

To search traces:

1. In the navigation menu select Explorers > Trace Explorer.

2. Use the Query builder to define the criteria of your search:

   • Choose the time period to display results for, which defaults to the last 15 minutes.

     When choosing a time window, selecting a period longer than one day greatly impacts load time for search results. For optimal results, scope search queries to shorter time periods.

   • Select the duration of traces (in seconds) to include in your search.

     Trace duration is calculated as the time elapsed from the earliest to the latest observed timestamps across all spans for a given trace.

   • Choose whether to include or exclude error states.

3. In the Span characteristics search bar, use the search query syntax to enter span criteria to match on, which can be a combination of values for service, operation, and tag.

   For example, the following query returns all traces where at least one span includes a service called billing-svc, an operation that starts with execute, and a tag named build.version where the value equals 1.54:

   service="billing-svc" operation=~"^execute*." tag:build.version="v1.54"

   You can also enter a unique trace ID to search for a specific trace, such as 19323ae9283b8fea000f63bff84524a0.

4. Click Run to submit your query. You must click Run after each time you modify the query criteria.

5. To display a natural language summary of your query, click Show summary. For example, the summary for the previous query displays as:

   All traces in the selected time range where at least one span has service "billing-svc" operation=~"^execute*." tag:build.version="v1.54.

   Click Hide summary to hide the search summary.

6. To refine your search, click + Add and specify additional span properties to include or exclude.

   Each row is a separate span filter that Observability Platform evaluates against individual spans. When you include multiple span filter rows, Observability Platform combines the filter rows as "and also" operations.

   Observability Platform finds traces where one span matches all criteria in the first row, and also where at least one other span matches all criteria in subsequent rows.

   As you modify your search, the search timeline graph updates in real time, along with the statistics, traces, and topology views. The timeline shows an overview of the current search, with counts of successful and failed traces plotted over time.

   

7. To copy a short URL to your clipboard you can share with other users, click Copy URL. Use this capability for sharing links of complicated queries that can generate long URLs.

   Observability Platform permanently stores short URLs in your tenant so that they don't expire.

8. To learn more about the spans in a trace, click the Traces tab. Click the name of a trace in the Trace column to open the trace details page, which displays information for each of the spans that comprise the trace.

   In some instances, a trace might contain a missing span. Observability Platform identifies these spans for the selected trace as Missing in the Stats panel. To search for traces containing a missing span, in the Span characteristics field in Trace Explorer, enter tag:parent_missing="true".

9. On the trace details page, filter the displayed data with the following options:

   • Use the Service and Operation menus to scope spans to specific services or operations.
   • Use the Only Errors toggle to display only segments of each trace that contain errors.
   • Use the Only Critical Path toggle to highlight segments of each span that impact the total duration of a trace. These segments help to identify latency issues within the trace. Resolving these issues can lead to the greatest improvement in latency of the entire trace.

10. In the Span details pane, use the quick copy button to copy the service name, operation name, or span ID, which you can enter in the Query builder of the Trace Explorer page.

    You can also hold the pointer over different attributes in the Span details pane, click the more icon , and select Add to filter to automatically navigate to the Trace Explorer page with that attribute included in your search criteria.

11. Use the following filters in the Query builder to limit search results to specific criteria:

    • To filter search results to the critical path, in the top navigation, click Trace Explorer. In the Span characteristics section of the Query builder, select Include and then enter critical_path="true". The search results update to display spans that can help to identify latency issues within the trace. Resolving these issues can lead to the greatest improvement in latency of the entire trace.

    • To filter search results to leaf errors, in the Span characteristics section of the Query builder, enter leaf_error="true". The search results update to display error spans that have no failing child spans. Leaf errors help filter out propagated errors, and provide clearer signals about the source of an error that might be causing the entire trace to fail.

12. To clear all selected options in the Query builder, click Clear.

Search query syntax

To define the properties you want to search for, you must use a particular search syntax. This syntax is a key followed by an operator and a value, surrounded by double quotes.

KEY =|=~ "VALUE"

In the Query builder section of the Trace Explorer page, click Syntax helper to display a dialog with syntax examples.

Use the following syntax to match specific properties:

Access recent searches

When investigating issues, you might run the same search frequently. Rather than redefining the search query, use recent searches to access previous searches. You can run a search from a previous time period and inspect the parameters of a previous search by clicking the search from the list.

For example, you can run a fully defined search query from one day ago for a particular time period. Queries run based on the relative time parameters from the recent search. If the selected time window of the query is the Last 15 minutes, the query pulls data from the last 15 minutes in current time, and not the last 15 minutes from when the search ran initially.

To access recent searches:

1. In the navigation menu select Explorers > Trace Explorer.

2. In the Search for Traces pane, click View searches.

3. Click the Recent tab, locate the search that you want to run, and click it.

   The parameters in the recent search override any trace or span information in the current Query builder.

Save frequent searches

If you find that you're accessing a recent search frequently, you can save the search so that it's always available to you. Saved searches are like bookmarks that you can reference whenever you need them.

To save a search:

1. In the navigation menu select Explorers > Trace Explorer.
2. Define the criteria of your search.
3. Click Save search.
4. In the Save search panel, enter a name for your search.
5. Click Save.

To view your saved search, click View searches. Your saved search displays in the Saved tab.

To share your search, click the copy icon to copy a short URL to your clipboard that you can share with other users.

Group and narrow results

After an initial search returns results, you can also group and narrow your results to specific attributes.

Use the Group by field to retrieve and display the aggregated results grouped by up to two different attributes, such as service and operation. This ability to group related attributes helps to highlight and better understand existing issues such as anomalous error rates, based on specific properties.

You can additionally narrow your results with the Narrow scope by service tag field. Enter a specific attribute to narrow your results to only that attribute. You can use this field independently or in conjunction with the Group by field. This field works with the Statistics table, Top Tags, and Topology view.

For example, you might have a payment app that includes a service named payment_gateway_service, and you want to explore issues with that service.

To group and narrow results:

1. In the Query builder, enter service="payment_gateway_service" to include that service

2. In the Group by menu, select Operation to group results by operation.

   In the Statistics table, you notice that the /paymentstore.PaymentStore/PaymentWasAuthorized operation has a high request rate. This operation tracks all authorized payments to your app.

3. In the Statistics table, click the /paymentstore.PaymentStore/PaymentWasAuthorized operation and then click Include in query to add that attribute to your query.

   Alternatively, you can click a service or operation and click Exclude from query to explicitly exclude that attribute from your query.

4. In the Narrow scope by service tag field, enter payment_gateway_service to focus your results to only that attribute.

   You want to view all credit card brands that users made payments with through the /paymentstore.PaymentStore/PaymentWasAuthorized operation.

5. In the Group by field, clear the Service option, and then enter card.brand to additionally group your results.

You identify the credit card brand that has the highest request rate among all cards that users make payments with, and can investigate and remediate related issues.