Query log data

Construct a query

Logs Explorer lets you query log data to focus your search. Begin by querying on one of the predefined keys, such as service or severity, and then use the query syntax to incorporate additional attributes.

Chronosphere recommends always including a filter for service in your query for optimal performance.

Basic querying

Use this method of selecting individual keys or values to query logs from a broad scope to a narrow focus. The following steps are recommended methods of querying. You can choose to start with a different key or value based on what you're searching for.

To use basic querying:

1. In the navigation menu select Explorers > Logs Explorer (Preview).

2. On the Logs Explorer page, select a time window to display logs for. The default time window is the last hour.

3. Define your query. You can use the sidebar, query box, or a combination of both to specify your query criteria. You can also click individual keys or values within a selected log.

   • Sidebar: Expand the key you want to query on, and click the value you want to show or hide matching logs for.

     For example, expand the severity key and click ERROR, which opens a dialog menu. From the menu, click Show matching logs to include that key/value combination in your query:

     severity = "ERROR"

   • Query box: Use the query syntax to enter the key you want to query on. The autocomplete syntax suggests operators and matching values for keys you enter as you type to help you construct your query.

     severity = "ERROR AND service = "gateway"
     If you're unsure what syntax to use, click in the query box and press Control+Space to display values for a key or available operators.

   • Attributes: After expanding an individual log, click any key or value to display a menu with the following options:

     • Show matching logs: Return logs containing only the selected key or value.
     • Hide matching logs: Return logs that don't contain the selected key or value.
     • Add to group and visualize: Adds the selected key or value to the query box as a summarize query, and selects a visualization that best matches the data type.
     • Add field to summary: Include the selected key or value in the Summary column of the log viewer.
     • Pin this field to top of field list: Pins the selected field to the top of the list of fields, which makes the field always available.
     • Copy value: Copy the selected key or value.

4. To submit your query, either click Run query or press Ctrl+Enter (Command+Return on macOS).

5. Expand your query by either adding additional key/value pairs or entering a full-text string such as "failed query token" to find logs that contain the expression anywhere in the log. For example:

   severity = "ERROR" and service = "gateway" AND "failed query token"
   The operators AND plus OR are case insensitive, so you can use AND, and, OR, and or interchangeably.

6. As you refine your filter, click and select a portion of the time chart to zoom in to view a smaller time window.

The results update to include only logs that contain the key/value pairs you enter.

Advanced querying

If you know details about the log you're searching for, or are carrying context to Logs Explorer from a services page, use the query syntax to construct your query.

For example, if you know there's an issue with the gateway service in your production environment, create a query to help you locate which Kubernetes cluster is experiencing issues.

To use advanced filtering:

1. In the navigation menu select Explorers > Logs Explorer (Preview).

2. In the query box, construct a query to include any logs with ERROR as the severity for the gateway service:

   service = "gateway" AND severity = "ERROR"

   The results include 8,700 logs.

3. To submit your query, click Run query or press Ctrl+Enter (Command+Return on macOS).

   You notice that several logs in the results contain "Failed to query user by token" in the Summary column.

4. Add a full-text string to your search to narrow the scope of your query, and then click Run query:

   service = "gateway" AND severity = "ERROR" AND "failed to query user by token"

   The results include 5,400 logs, which is fewer, but still too many.

   In the sidebar, you notice that 75% of the results are for the production-east Kubernetes cluster.

5. In the sidebar, click production-east and then click Show matching logs. Observability Platform adds the selected key/value to your query:

   service = "gateway" AND severity = "ERROR" AND "failed to query user by token"
   AND kubernetes.cluster = "production-east"

   The results include 4,000 logs. To reduce scope, begin drilling in to individual logs.

6. Expand individual logs to find commonalities across the data. You realize that the same Kubernetes pod is included in many of the logs, so you add that key/value pair to your filter:

   service = "gateway" AND severity = "ERROR" AND "failed to query user by token"
   AND kubernetes.cluster = "production-east"
   AND kubernetes.pod_name = "gateway-6agg9df321-o89ef"

   The results include less than 700 logs, which is 8,000 fewer than your initial query.

You identified the individual Kubernetes pod containing the majority of errors for the gateway service so you can inform your team and begin fixing the issue.

Group and visualize queries

When exploring log data, you might want to group logs by one or more fields. This capability helps to answer questions such as, "What are my error rates across environments for my service?".

Observability Platform lets you group logs and visualize the results within Logs Explorer. You can explicitly use the summarize operator in a query, select a key or value in the sidebar, or choose attributes in an individual log to group your query results by.

To group and visualize query results:

1. In the navigation menu select Explorers > Logs Explorer (Preview).

2. Select a field or value to group by, or create a query with the summarize operator:

   • Sidebar: In the sidebar list of fields, expand the key you want to group by, hover the pointer over a value, click the three vertical dots icon, and then click Add to group and visualize.

   • Attributes: Expand an individual log, click any key or value, and then click Add to group and visualize.

   • Query box: Use the query syntax to create a query that includes the summarize operator. For example, the following query searches for logs in a Kubernetes cluster in production-us-west that contain errors, and then groups the results by service and environment:

     severity = "ERROR"
     AND kubernetes.cluster_name = "production-us-west"
     | summarize by service, environment

   Logs Explorer inserts the key or value you selected into a summarize query and visualizes the results.

3. To display the results with a different visualization, select one of the available options. For example, select Bar to visualize the results as a bar chart.

Save and share queries

You can save queries that you run frequently, share queries, and share links to individual logs to help focus results when investigating issues.

Access recent and saved queries

When investigating issues, you might use the same query frequently. Rather than redefining the query, use recent and saved queries to access previously defined queries in Observability Platform. You can apply a fully defined query from a previous time period by clicking a query from the Examples tab.

Recent query are available globally to all users in Observability Platform and persist for 14 days.

To access recent and saved queries:

1. In the navigation menu select Explorers > Logs Explorer (Preview).
2. Click View queries to display all available queries.
3. Click the Recent tab or the Saved tab to display the queries you want to view.
4. Locate the query you want to apply and click it.

The parameters in the query override any parameters in the query box.

Save a query

You can save queries that you access frequently so they're always available in Observability Platform. Saved queries are like bookmarks you can reference when you need them.

To save a query:

1. In the navigation menu select Explorers > Logs Explorer (Preview).
2. In the query box, construct your query. The Save query button is unavailable until you run your query.
3. Click Run query to run your query.
4. Click Save query to save your query.
5. In the Queries window, enter a name for your query and click Save.

Your query displays in the Saved tab of the Queries window. You can access your saved queries and apply them at any time.

Share a URL to a query

When investigating issues, you might want to share a defined query with other users, or include a URL to a defined query in monitor annotations, runbooks, or other on-call tools. Logs Explorer lets you copy a short URL to a defined query using either a relative or absolute time range.

Relative time is useful for understanding the results of a query in a past period of time relative to the current time, such as in the past 30 minutes. Absolute time is better suited for comparing results across a fixed point in time, such as the results of a query from last Monday at 8:00 AM versus that same query run today.

To copy a URL to a defined query:

1. In the navigation menu select Explorers > Logs Explorer (Preview).

2. Construct a query that returns the log data you want to view.

3. In the main header, click Copy URL and choose type of time range to use for the URL:

   • Copy with absolute time range: Create a link that runs a query against the date and time interval from when you copied the link. For example, if your query uses 1h as the time interval, the time range is exactly one hour ago, based on the date and time you ran the query.

   • Copy with relative time range: Create a link that runs a query against the current time. For example, if your query uses 1h as the time interval, the time range is exactly one hour ago from the current time.

The URL is copied to your clipboard based on your selection.

Share a URL to a specific log

When troubleshooting issues, you might want to share a link to a specific log with other users, rather than to a defined query.

To copy a URL to a specific log:

1. In the navigation menu select Explorers > Logs Explorer (Preview).
2. Construct a query that returns the log data you want to view.
3. Expand an individual log you want to share, and then click Share log to copy a URL to the selected log.

The URL is compressed to a short URL and copied to your clipboard. When another user opens this URL, the results in Logs Explorer focus on the selected log only.

Logging query syntax

Use the logging query language within Logs Explorer to construct search queries for retrieving, processing, and analyzing your log data.

Observability Platform provides a lightweight, flexible syntax for querying log data in Logs Explorer. This syntax implements predefined keys that accept a comparison operator, such as an equals sign =, and a value.

KEY =|!=|=~|!~|: VALUE AND|OR (KEY =|!=|=~|!~|: VALUE) AND|OR "full-text search value"
AND|OR KEY EXISTS

The query language supports full-text search using double quotes ("") to find logs that contain the expression anywhere in the log. Longer, more specific full-text searches are more optimized and return results faster.

The following query matches any service named gateway where the kubernetes.namespace_name is test-logging or kubernetes.cluster_name is test and the log message contains "insert success":

service = "gateway" AND (kubernetes.namespace_name = "test-logging" OR
kubernetes.cluster_name = "test") AND "insert success"

Features

The Observability Platform query syntax for logging supports the following features:

• Autocomplete: Start typing a key to get autocomplete values. Press Control+Space to display suggestions.
• Run shortcut: Run a query by pressing Control+Enter (Command+Return on macOS).
• Nested queries: Use parentheses () to establish the order of operations for complex queries.
• Field search: Query on specific fields by entering a key/value pair such as key = "value".
• String identification: Use either single '' or double "" quotes to identify strings.
• Full-text search: Express a single value expression, such as "query user token", to find logs that contains the expression anywhere in the log. Full-text searches must be surrounded by double quotes.
• Exists: Find logs with a specific key by entering key.label EXISTS to return any logs containing that key and label combination. For example, kubernetes.namespace_name EXISTS returns any logs containing kubernetes.namespace_name.

Keys

The query syntax supports the following keys:

• service: Services that Observability Platform discovers in your log data. Chronosphere recommends always including a filter for service in your query for optimal performance.
• severity: Severity of issues, sanitized to the following values: DEBUG, INFO, WARN, ERROR, FATAL.
• message: Human-readable description of the log, derived from your log data.
• Custom labels you created for your logs.

Operators

After entering an operator in the query box, you can click the operator to display a dropdown with additional supported operators. This feature lets you quickly change operators in your existing query. If you modify operators with this feature, you must click Run query or use the keyboard shortcut Control+Return (Command+Enter on macOS) to submit the modified query.

The querying syntax supports the following operators:

Queries with the AND operator take precedence. If your query doesn't use parentheses, Observability Platform evaluates all AND statements sequentially, followed by any OR statements and the next set of AND queries.

The operators AND plus OR are case insensitive, so you can use AND, and, OR, and or interchangeably.

You can also use mathematical operators for addition (+), subtraction (-), multiplication (*), and division (/) to complete calculations within a query. This capability is especially useful when creating monitors to generate alerts and receive notifications. For example, the following query includes a calculation for an alert that returns a count() if any logs contain severity="ERROR":

kubernetes.cluster_name =~ "production_*"
| summarize (countif(severity="ERROR")/count()) by service

Use parentheses to nest calculations and separate them from the remainder of the search query.

Query examples

This syntax supports nesting using parentheses so you can create complex queries. For example, the following query matches on two severity values, or where the message contains “database connection” and the mysql.table_name label starts with production.

service = "gateway" AND severity =~ "WARN|ERROR" OR (message : "database connection"
AND mysql.table_name =~ "^production.*")

The following query matches on a service named gateway where the kubernetes.pod_id equals 3bf26945-b817-4e72-b22c-662c318af2f1 anywhere in the log payload.

service = "gateway" AND kubernetes.pod_id = "3bf26945-b817-4e72-b22c-662c318af2f1"

The following query returns all logs for a service named gateway that have a logger other than deleter, or have no logger set.

SERVICE = "gateway" AND NOT logger = "deleter"

Transformation operators

Transformation queries include operators that modify input records by adding, removing, or updating fields in the results set. Observability Platform supports the following transformation operators you can include in log queries.

All transformation operators require a pipe (|) character in a query.

limit

The limit operator returns the most recent log events.

Syntax

[query] | limit [limit]

Arguments

Example

severity = "WARNING"
| limit 500

make-series

The make-series operator shapes logs to include in a chart with a time X-axis. Use this operator to visualize log queries containing time series data. The operator sorts multiple time series in descending order by the last data point value.

Syntax

[query] | make-series [aggregation] [step] by [GroupExpression]

Arguments

Example

severity = "WARNING"
| make-series avg(latencyInSeconds) step 15m by severity, service

sort

The sort operator sorts your results by a specified column, in either ascending or descending order. The operator supports sorting by multiple columns, such as service and severity. When sorting by multiple columns, you can specify the sort order for each column, as shown in the following example.

Syntax

[query] | sort by column [asc | desc] [, column ...]

Arguments

Example

severity = "WARNING"
| summarize count() by service, severity
| sort by count_ desc, service asc
| limit 10

summarize

The summarize operator collects logs with identical by columns. The operator sorts grouped data in descending order.

Syntax

[query] | summarize [aggregation] by [GroupExpression]

Arguments

Example

severity = "ERROR"
| summarize sum(production) by severity, service
| limit 100

top-nested

The top-nested operator completes aggregation and value selection in a hierarchical manner.

The operator then partitions data based on criteria from the first top-nested clause. Then, the operator selects the top records in each partition using an aggregation (if specified). Subsequent top-nested clauses refine the partitions from the previous clause, creating a hierarchy of more precise groupings.

This operator creates a table with two columns for each clause. One column contains unique values from the expression, and the other column shows the results obtained from the aggregation calculation.

Syntax

[query] | top-nested [ N ] of [expression] by [aggregation] [asc | desc] [, top-nested ... ]

Arguments

Example

severity = "ERROR" AND cluster.name = "production"
| top-nested 10 of service by avg(duration)

Aggregation functions

Aggregation queries include functions that can combine their input into a new structure, or emit new events into the output stream. You can use all aggregate functions in summarize and make-series operators. Observability Platform supports the following aggregation functions you can include in log queries.

avg()

The avg function computes the average of values of an expression over events passing through the function. This function only works with fields containing numeric values.

Syntax

avg(expression)

Arguments

Example

key1 = 'value1'
| summarize avg(retrycount) by severity

count()

The count function returns a count of the number of events passing through the function.

Syntax

count()

Arguments

None.

Example

key1 = 'value1'
| make-series count() by severity
| limit 100

key1 = 'value1'
| summarize count() by severity
| limit 100

dcount()

The dcount function enumerates the number of distinct values of an expression in events passing through the function.

Syntax

dcount(expression)

Arguments

Example

key1 = 'value1'
| make-series dcount(hostname) by severity

percentile()

The percentile function returns the specified percentile value of an expression. This function only works with fields containing numeric values.

Syntax

percentile(expression, percentile)

Arguments

Example

key1 = 'value1'
| make-series percentile(duration, 95) by severity

sum()

The sum function computes the sum of values of an expression over events passing through the function.

Syntax

sum(expression)

Arguments

Example

key1 = 'value1'
| summarize sum(retrycount) by severity