Logging query syntax

Use the logging query language within Logs Explorer to construct search queries for retrieving, processing, and analyzing your log data.

Observability Platform provides a lightweight, flexible syntax for querying log data in Logs Explorer. This syntax implements predefined keys that accept a comparison operator, such as an equals sign =, and a value.

KEY =|!=|=~|!~|: VALUE AND|OR (KEY =|!=|=~|!~|: VALUE) AND|OR "full-text search value"
AND|OR KEY EXISTS

The query language supports full-text search using double quotes ("") to find logs that contain the filter expression anywhere in the log. Longer, more specific full-text searches are more optimized and return results faster.

Features

The query syntax for Logs Explorer supports the following features:

• Autocomplete: Start typing a key to get autocomplete values. Press Control+Space to display suggestions.
• Run shortcut: Run a query by pressing Control+Enter (Command+Return on macOS).
• Nested queries: Use parentheses () to establish the order of operations for complex queries.
• Field search: Query on specific fields by entering a key/value pair such as key = "value".
• String identification: Use either single '' or double "" quotes to identify strings.
• Full-text search: Express a single value filter expression, such as "query user token", to find logs that contains the filter expression anywhere in the log. Full-text searches must be surrounded by double quotes.
• Exists: Find logs with a specific key by entering key.label EXISTS to return any logs containing that key and label combination. For example, kubernetes.namespace_name EXISTS returns any logs containing kubernetes.namespace_name.

Keys

Log Explorer supports querying all user-defined keys in your data, in addition to the following derived keys:

• service: Services that Observability Platform discovers in your log data. Chronosphere recommends always including a filter for service in your query for optimal performance.
• severity: Severity of issues, sanitized to the following values: DEBUG, INFO, WARN, ERROR, FATAL.
• message: Human-readable description of the log, derived from your log data.
• Custom labels you created for your logs.

Operators

The querying syntax for Logs Explorer supports the following operators:

• Logical operators
• Mathematical operators
• Transformation operators

Logical operators

The querying syntax for Logs Explorer supports the following logical operators:

Queries with the AND operator take precedence. If your query doesn't use parentheses, Observability Platform evaluates all AND statements sequentially, followed by any OR statements and the next set of AND queries.

To use AND plus OR operators in the same query, separate them with parentheses. For example, the following query matches any service named gateway where the kubernetes.namespace_name is test-logging or kubernetes.cluster_name is test and the log message contains "insert success":

service = "gateway" AND (kubernetes.namespace_name = "test-logging" OR
kubernetes.cluster_name = "test") AND "insert success"

The operators AND plus OR are not case sensitive, so you can use AND, and, OR, and or interchangeably.

Mathematical operators

You can also use mathematical operators for addition (+), subtraction (-), multiplication (*), and division (/) to complete calculations within a query.

The querying syntax supports the following mathematical operators:

This capability is especially useful when creating monitors to generate alerts and receive notifications. For example, the following query includes a calculation for an alert that returns the ratio of logs without errors to total logs in the nginx service:

service = "nginx"
| summarize 1 - countif(severity="ERROR")/countif(severity EXISTS)

Use parentheses to nest calculations and separate them from the remainder of the search query.

Transformation operators

Transformation queries include operators that modify input records by adding, removing, or updating fields in the results set. Observability Platform supports the following transformation operators you can include in log queries.

limit

The limit operator returns the most recent log events.

Syntax

QUERY | limit NUMBER

Arguments

Example

severity = "WARNING"
| limit 500

make-series

The make-series operator shapes logs to include in a time chart with an X-axis. Use this operator to visualize log queries containing time series data. The operator sorts multiple time series in descending order by the last data point value.

Syntax

QUERY | make-series AGGREGATION step TIME by EXPRESSION

Arguments

Example

severity = "WARNING"
| make-series avg(latencyInSeconds) step 15m by severity, service

sort

The sort operator sorts your results by a specified column, in either ascending or descending order. The operator supports sorting by multiple columns, such as service and severity. When sorting by multiple columns, you can specify the sort order for each column, as shown in the following example.

Syntax

QUERY | sort by COLUMN asc|desc [, COLUMN ...]

Arguments

Example

severity = "WARNING"
| summarize count() by service, severity
| sort by count_ desc, service asc
| limit 10

summarize

The summarize operator generates a table of data that aggregates the content of the table from the input query, grouped in descending order. By default, this operator uses the count() function to count all permutations of the specified field if no AGGREGATION is specified.

Input rows are arranged into groups that have the same expression. Use the by keyword to specify the columns you want to group your data by before applying any aggregations.

Your query can include only one summarize operator. To group data and refine the results with subsequent clauses, use the top-nested operator.

Syntax

QUERY | summarize AGGREGATION by FIELD EXPRESSION

Arguments

Example

severity = "ERROR"
| summarize sum(production) by severity, service
| limit 100

top-nested

The top-nested operator completes aggregation and value selection in a hierarchical manner. The operator partitions data based on criteria from the first top-nested clause, and then selects the top records in each partition using an aggregation, if specified.

Unlike the summarize operator, you can include multiple top-nested clauses in a single query. Using a single pipe (|) separator, specify a "root" top-nested clause, and then nest additional top-nested clauses separated by a comma. Each of these clauses refine the partitions from the previous clause, creating a hierarchy of more precise groupings.

For example, using the summarize operator, the following query returns the count of logs from each endpoint from the nginx-requests service in descending order:

service = "nginx-requests"
| summarize by http.response.endpoints

The summarize operator can use additional aggregation functions, but on its own returns a two-column table containing a count of the specified field. The top-nested operator is similar, but provides additional options to partition and display results.

The top-nested operator creates a table with two columns for each clause. One column contains unique values from the filter EXPRESSION, and the other column shows the results obtained from the AGGREGATION calculation.

Using the top-nested operator, the following query refines the results from the nginx-requests service to the top 80 endpoints, and further partitions the data by the top nine HTTP response status codes:

service = "nginx-requests"
| top-nested 80 of http.response.endpoints,
top-nested 9 of http.response.status_code

This query returns a four-column table with a row for each entry that includes:

• The name of the endpoint from http.response.endpoints.
• The aggregated value for the endpoint.
• The HTTP status code from http.response.status_code.
• The aggregated status code for the endpoint.

The number of rows returned by this query is calculated by multiplying the number of values returned for http.response.endpoints times the number of values returned for http.response.status_code.

Syntax

QUERY | top-nested NUMBER of EXPRESSION by AGGREGATION asc|desc [, top-nested ...]

Arguments

Example

severity = "ERROR" AND cluster_name = "production"
| top-nested 10 of service by avg(duration)

severity = "ERROR" AND cluster_name = "production"
| top-nested 5 of service, top-nested of kubernetes_namespace

Regular expressions

Logs Explorer uses re2 (opens in a new tab) regular expression syntax. The following regular expressions are supported, and can be used between fields and values:

In the following example, consider that a field name kubernetes.cluster_name contains the following values: production-1, production-2, production-3, productionNEW, old_production.

The following query matches the word production anywhere in the kubernetes.cluster_name field, so it matches all values in the field:

kubernetes.cluster_name =~ "production"

The following regular expression query includes a period and wildcard (.*), which matches the word production, followed by zero or more characters anywhere in the phrase. This query also matches all values in the field:

kubernetes.cluster_name =~ "production.*"

Using a dash and wildcard (-*) matches the word production, followed by zero or more dash characters anywhere in the phrase. This query also matches all values in the field, including productionNEW and old_production, which might not be immediately apparent:

kubernetes.cluster_name =~ "production-*"

To match only the fields containing production-, combine the dash and period characters with a wildcard and add a dollar sign ($). The following query matches only production-1, production-2, production-3, but does not match either productionNEW or old_production:

kubernetes.cluster_name =~ "production-.*$"

To match a substring only, use a colon (:). The following query matches any logs where the kubernetes.cluster_name field contains only production-us-east:

kubernetes.cluster_name: "production-us-east"

Arrays

You can query array values up to the first level. During ingestion Observability Platform flattens multiple level arrays to one level to preserve order. If a key precedes the array index, the index must always be at the end.

For example, consider the following array of objects belonging to the nginx service as they display in Logs Explorer:

tags:
    0:
	      application: a
	      events: b
	      ingestion: c
    1:
	      application: d
	      events: e
	      ingestion: f

To return results in the second array position (1) that match events = e, use the following query:

service = "nginx"
tags.events[1] = "e"

Query examples

This syntax supports nesting using parentheses so you can create complex queries. For example, the following query matches on two severity values, or where the message contains “database connection” and the mysql.table_name label starts with production:

service = "gateway" AND severity =~ "WARN|ERROR" OR (message : "database connection"
AND mysql.table_name =~ "^production.*")

The following query matches on a service named gateway where the kubernetes.pod_id equals 3bf26945-b817-4e72-b22c-662c318af2f1 anywhere in the log payload:

service = "gateway" AND kubernetes.pod_id = "3bf26945-b817-4e72-b22c-662c318af2f1"

The following query returns all logs for a service named gateway that have a logger other than deleter, or have no logger set:

SERVICE = "gateway" AND NOT logger = "deleter"

Aggregation functions

Aggregation queries include functions that can combine their input into a new structure, or emit new events into the output stream. You can use all aggregation functions with summarize and make-series operators. Observability Platform supports the following aggregation functions you can include in log queries.

Group results

To group results in queries that include an aggregation function, use the by operator. This operator lets you group results by a specified field, which is different than the sort operator, which lets you sort results by a specified column.

For example, the following query includes the summarize transformation operator to return the average for the httpRequest.responseSize field, and then groups the results by service:

service = "nginx"
| summarize avg(httpRequest.responseSize) by service

Alias function names

When running a query containing a transformation operator and an aggregation function, you can add an alias for the function name that displays in the generated visualization. This capability applies to all transformation operators and aggregation functions.

For example, the following query creates a table of data that aggregates the content of the table from the input query, and displays the count by service. The column header that contains the total count is named _count.

severity = "ERROR"
| summarize count() by service

The following query returns the same data, but uses an alias named total for the count() function. The column header containing the total count is named total.

severity = "ERROR"
| summarize total = count() by service

Similarly, you can create an alias using the make-series operator:

severity = "WARN"
| make-series total = count() by service

avg()

The avg function computes the average of values of a filter EXPRESSION over events passing through the function. This function only works with fields containing numeric values.

Syntax

QUERY | OPERATOR avg(EXPRESSION)

Arguments

Example

service = "nginx"
| summarize avg(httpRequest.responseSize) by service

avgif()

The avgif function computes the average of values of a field where the specified filter EXPRESSION evaluates to true. This function works only with fields containing numeric values.

Syntax

QUERY | OPERATOR avgif(FIELD, EXPRESSION)

Arguments

Example

service = "nginx"
| summarize avgif(httpRequest.responseSize, httpRequest.responseSize > 2000) by service

count()

The count function returns a count of the number of events passing through the function.

Syntax

QUERY | OPERATOR count()

Arguments

Example

service = "checkout-service"
| make-series count() by severity
| limit 100

countif()

The countif function returns a count of rows where the specified filter EXPRESSION evaluates to true. This function ignores null values.

Syntax

QUERY | OPERATOR countif(EXPRESSION)

Arguments

Example

service = "nginx"
| summarize countif(kubernetes.pod_name =~ "nginx-*") by severity

dcount()

The dcount function calculates an estimated number of distinct values of a FIELD in events passing through the function.

Syntax

QUERY | OPERATOR dcount(FIELD)

Arguments

Example

service = "nginx"
| make-series dcount(hostname) by severity

dcountif()

The dcountif function calculates an estimated number of distinct values where the specified EXPRESSION filter evaluates to true. Use this function to estimate the cardinality of large data sets.

Syntax

QUERY | OPERATOR dcountif(FIELD, EXPRESSION)

Arguments

Example

service = "nginx"
| summarize dcountif(kubernetes.pod_name, kubernetes.pod_name =~ "nginx-*") by "severity"

min()

The min function returns the minimum value of the specified field.

Syntax

QUERY | OPERATOR min(FIELD)

Arguments

Example

service = "nginx"
| summarize min(httpRequest.responseSize) by kubernetes.cluster_name

minif()

The min function returns an estimate of the minimum value of the specified field where the EXPRESSION filter evaluates to true.

Syntax

QUERY | OPERATOR minif(FIELD, EXPRESSION)

Arguments

Example

service = "nginx"
| summarize minif(httpRequest.responseSize, httpRequest.responseSize > 100) by kubernetes.cluster_name

max()

The max function returns the maximum value of the specified field.

Syntax

QUERY | OPERATOR min(FIELD)

Arguments

Example

service = "nginx"
| summarize max(httpRequest.responseSize) by "kubernetes.cluster_name"

maxif()

The maxif function returns an estimate of the maximum value of the specified field where the EXPRESSION filter evaluates to true.

Syntax

QUERY | OPERATOR maxif(FIELD, EXPRESSION)

Arguments

Example

service = "nginx"
| summarize maxif(httpRequest.responseSize, httpRequest.responseSize > 100) by kubernetes.cluster_name

percentile()

The percentile function returns the specified percentile value of a filter EXPRESSION. This function only works with fields containing numeric values.

Syntax

QUERY | OPERATOR percentile(EXPRESSION, NUMBER)

Arguments

Example

key1 = 'value1'
| make-series percentile(duration, 95) by severity

substring()

The substring function extracts a substring from the source string based on a starting index character position.

Syntax

QUERY | OPERATOR by substring(FIELD, START_INDEX, LENGTH)

Arguments

Example

The following query includes a colon (:) to find all logs that have a resource.type field containing a k8s substring. Then, the query groups the logs by the resource.type field, groups the logs by the first 13 characters of the k8s substring, and returns those groups.

resource.type: "k8s"
| summarize by substring(resource.type, 0, 13)

In the sidebar, expanding resource.type shows that this query matches values such as k8s_container, k8s_resources, and k8s_cluster.

The following query uses the substring() function as a filter without any aggregation. This kind of query is useful when you want to ensure that a particular substring matches a value. In this case, the query matches any logs containing writer in the kubernetes.pod_name field:

substring(kubernetes.pod_name, 3, 6) = "writer"

sum()

The sum function computes the sum of values of a filter EXPRESSION over events passing through the function.

Syntax

QUERY | OPERATOR sum(EXPRESSION)

Arguments

Example

kubernetes.cluster =~ "^production-*"
| summarize sum(destination.bytes) by location-country

sumif()

The sumif function computes the sum of values for the specified field where the EXPRESSION filter evaluates to true.

Syntax

QUERY | OPERATOR sumif(FIELD, EXPRESSION)

Arguments

Example

kubernetes.cluster =~ "^production-*"
| summarize sumif(destination.bytes, destination.bytes > 1000) by location-country

sum(_payloadSize)

The sum(_payloadSize) function calculates the size of the log payload in bytes.

Syntax

QUERY | OPERATOR sum(_payloadSize) by FIELD

Arguments