Query log data

Construct a query

Logs Explorer lets you query log data to focus your search. Begin by querying on one of the predefined keys, such as service or severity, and then use the query syntax to incorporate additional attributes.

Chronosphere recommends always including a filter for service in your query for optimal performance.

Basic querying

Use this method of selecting individual keys or values to query logs from a broad scope to a narrow focus. The following steps are recommended methods of querying. You can choose to start with a different key or value based on what you're searching for.

To use basic querying:

1. In the navigation menu select Explorers > Logs Explorer (Preview).

2. On the Logs Explorer page, select a time window to display logs for. The default time window is the last hour.

3. Define your query. You can use the sidebar, query box, or a combination of both to specify your query criteria. You can also click individual keys or values within a selected log.

   • Sidebar: Expand the key you want to query on, and click the value you want to show or hide matching logs for.

     For example, expand the severity key and click ERROR, which opens a dialog menu. From the menu, click Show matching logs to include that key/value combination in your query:

     severity = "ERROR"

   • Query box: Use the query syntax to enter the key you want to query on. The autocomplete syntax suggests operators and matching values for keys you enter as you type to help you construct your query.

     severity = "ERROR AND service = "gateway"
     If you're unsure what syntax to use, click in the query box and press Control+Space to display values for a key or available operators.

   • Attributes: After expanding an individual log, click any key or value to display a menu with the following options:

     • Show matching logs: Return logs containing only the selected key or value.
     • Hide matching logs: Return logs that don't contain the selected key or value.
     • Add to group and visualize: Adds the selected key or value to the query box as a summarize query, and selects a visualization that best matches the data type.
     • Add field to summary: Include the selected key or value in the Summary column of the log viewer.
     • Pin this field to top of field list: If you select a field in the sidebar, this option pins the selected field to the top of the list of fields, which makes the field always visible.
     • Add field as column: Adds the selected field as a column in the query results.
     • Copy value: Copy the selected key or value.

4. To submit your query, either click Run query or press Control+Enter (Command+Return on macOS).

5. Expand your query by either adding additional key/value pairs or entering a full-text string such as "failed query token" to find logs that contain the expression anywhere in the log. For example:

   severity = "ERROR" and service = "gateway" AND "failed query token"
   The operators AND plus OR are case insensitive, so you can use AND, and, OR, and or interchangeably.

6. As you refine your filter, click and select a portion of the time chart to zoom in to view a smaller time window.

The results update to include only logs that contain the key/value pairs you enter.

Advanced querying

If you know details about the log you're searching for, or are carrying context to Logs Explorer from a services page, use the query syntax to construct your query.

For example, if you know there's an issue with the gateway service in your production environment, create a query to help you locate which Kubernetes cluster is experiencing issues.

To use advanced filtering:

1. In the navigation menu select Explorers > Logs Explorer (Preview).

2. In the query box, construct a query to include any logs with ERROR as the severity for the gateway service:

   service = "gateway" AND severity = "ERROR"

   The results include 8,700 logs.

3. To submit your query, click Run query or press Control+Enter (Command+Return on macOS).

   You notice that several logs in the results contain "Failed to query user by token" in the Summary column.

4. Add a full-text string to your search to narrow the scope of your query, and then click Run query:

   service = "gateway" AND severity = "ERROR" AND "failed to query user by token"

   The results include 5,400 logs, which is fewer, but still too many.

   In the sidebar, you notice that 75% of the results are for the production-east Kubernetes cluster.

5. In the sidebar, click production-east and then click Show matching logs. Observability Platform adds the selected key/value to your query:

   service = "gateway" AND severity = "ERROR" AND "failed to query user by token"
   AND kubernetes.cluster = "production-east"

   The results include 4,000 logs. To reduce scope, begin drilling in to individual logs.

6. Expand individual logs to find commonalities across the data. You realize that the same Kubernetes pod is included in many of the logs, so you add that key/value pair to your filter:

   service = "gateway" AND severity = "ERROR" AND "failed to query user by token"
   AND kubernetes.cluster = "production-east"
   AND kubernetes.pod_name = "gateway-6agg9df321-o89ef"

   The results include less than 700 logs, which is 8,000 fewer than your initial query.

You identified the individual Kubernetes pod containing the majority of errors for the gateway service so you can inform your team and begin fixing the issue.

Group and visualize queries

When exploring log data, you might want to group logs by one or more fields. This capability helps to answer questions such as, "What are my error rates across environments for my service?".

Observability Platform lets you group logs and visualize the results within Logs Explorer. You can explicitly use the summarize operator in a query, select a key or value in the sidebar, or choose attributes in an individual log to group your query results by.

To group and visualize query results:

1. In the navigation menu select Explorers > Logs Explorer (Preview).

2. Select a field or value to group by, or create a query with the summarize operator:

   • Sidebar: In the sidebar list of fields, expand the key you want to group by, hover the pointer over a value, click the three vertical dots icon, and then click Add to group and visualize.

   • Attributes: Expand an individual log, click any key or value, and then click Add to group and visualize.

   • Query box: Use the query syntax to create a query that includes the summarize operator. For example, the following query searches for logs in a Kubernetes cluster in production-us-west that contain errors, and then groups the results by service and environment:

     severity = "ERROR"
     AND kubernetes.cluster_name = "production-us-west"
     | summarize by service, environment

   Logs Explorer inserts the key or value you selected into a summarize query and visualizes the results.

3. To display the results with a different visualization, select one of the available options. For example, select Bar to visualize the results as a bar chart.

Save and share queries

You can save queries that you run frequently, share queries, and share links to individual logs to help focus results when investigating issues.

Access recent and saved queries

When investigating issues, you might use the same query frequently. Rather than redefining the query, use recent and saved queries to access previously defined queries in Observability Platform. You can apply a fully defined query from a previous time period by clicking a query from the Examples tab.

Recent query are available globally to all users in Observability Platform and persist for 14 days.

To access recent and saved queries:

1. In the navigation menu select Explorers > Logs Explorer (Preview).
2. Click View queries to display all available queries.
3. Click the Recent tab or the Saved tab to display the queries you want to view.
4. Locate the query you want to apply and click it.

The parameters in the query override any parameters in the query box.

Save a query

You can save queries that you access frequently so they're always available in Observability Platform. Saved queries are like bookmarks you can reference when you need them.

To save a query:

1. In the navigation menu select Explorers > Logs Explorer (Preview).
2. In the query box, construct your query. The Save query button is unavailable until you run your query.
3. Click Run query to run your query.
4. Click Save query to save your query.
5. In the Queries window, enter a name for your query and click Save.

Your query displays in the Saved tab of the Queries window. You can access your saved queries and apply them at any time.

Share a URL to a query

When investigating issues, you might want to share a defined query with other users, or include a URL to a defined query in monitor annotations, runbooks, or other on-call tools. Logs Explorer lets you copy a short URL to a defined query using either a relative or absolute time range.

Relative time is useful for understanding the results of a query in a past period of time relative to the current time, such as in the past 30 minutes. Absolute time is better suited for comparing results across a fixed point in time, such as the results of a query from last Monday at 8:00 AM versus that same query run today.

To copy a URL to a defined query:

1. In the navigation menu select Explorers > Logs Explorer (Preview).

2. Construct a query that returns the log data you want to view.

3. In the main header, click Copy URL and choose type of time range to use for the URL:

   • Copy with absolute time range: Create a link that runs a query against the date and time interval from when you copied the link. For example, if your query uses 1h as the time interval, the time range is exactly one hour ago, based on the date and time you ran the query.

   • Copy with relative time range: Create a link that runs a query against the current time. For example, if your query uses 1h as the time interval, the time range is exactly one hour ago from the current time.

The URL is copied to your clipboard based on your selection.

Share a URL to a specific log

When troubleshooting issues, you might want to share a link to a specific log with other users, rather than to a defined query.

To copy a URL to a specific log:

1. In the navigation menu select Explorers > Logs Explorer (Preview).
2. Construct a query that returns the log data you want to view.
3. Expand an individual log you want to share, and then click Share log to copy a URL to the selected log.

The URL is compressed to a short URL and copied to your clipboard. When another user opens this URL, the results in Logs Explorer focus on the selected log only.