Query log data (native logging)

Construct a query

Logs Explorer lets you query log data to focus your search. Begin by querying on one of the predefined keys, such as service or severity, and then use the query syntax to incorporate additional attributes.

Chronosphere recommends always including a filter for service in your query for optimal performance.

Basic querying

Use this method of selecting individual keys or values to query logs from a broad scope to a narrow focus. The following steps are recommended methods of querying. You can choose to start with a different key or value based on what you're searching for.

To use basic querying:

1. In the navigation menu select Explorers > Logs Explorer (Preview).

2. On the Logs Explorer page, select a time window to display logs for. The default time window is the last hour.

3. Define your query. You can use the sidebar, query box, or a combination of both to specify your query criteria. You can also click individual keys or values within a selected log.

   • Sidebar: Expand the key you want to query on, and click the value you want to show or hide matching logs for.

     For example, expand the severity key and click ERROR, which opens a dialog menu. From the menu, click Show matching logs to include that key/value combination in your query:

     severity = "ERROR"

   • Query box: Use the query syntax to enter the key you want to query on. The autocomplete syntax suggests operators and matching values for keys you enter as you type to help you construct your query.

     severity = "ERROR AND service = "gateway"
     If you're unsure what syntax to use, click in the query box and press Control+Space to display values for a key or available operators.

   • Attributes: After expanding an individual log, click any key or value to display a menu with the following options:

     • Show matching logs: Return logs containing only the selected key or value.
     • Hide matching logs: Return logs that don't contain the selected key or value.
     • Add field to summary: Include the selected key or value in the Summary column of the log viewer.
     • Copy value: Copy the selected key or value.

4. To submit your query, either click Run query or press Windows+Enter (Command+Return on macOS).

5. Expand your query by either adding additional key/value pairs or entering a full text string such as “36z9aaebqcc4oosuwr3v97qrgp” to find logs that contain the expression anywhere in the log. For example:

   severity = "ERROR" and service = "gateway" AND “36z9aaebqcc4oosuwr3v97qrgp”
   The operators AND plus OR are case insensitive, so you can use AND, and, OR, and or interchangeably.

6. As you refine your filter, click and select a portion of the time chart to zoom in to view a smaller time window.

The results update to include only logs that contain the key/value pairs you enter.

Advanced querying

If you know details about the log you're searching for, or are carrying context to Logs Explorer from a services page, use the query syntax to construct your query.

For example, if you know there's an issue with the gateway service in your production environment, create a query to help you locate which Kubernetes cluster is experiencing issues.

To use advanced filtering:

1. In the navigation menu select Explorers > Logs Explorer (Preview).

2. In the query box, construct a query to include any logs with ERROR as the severity for the gateway service:

   service = "gateway" AND severity = "ERROR"

   The results include 8,700 logs.

3. To submit your query, click Run query or press Windows+Enter (Command+Return on macOS).

   You notice that several logs in the results contain Failed to query user by token in the Summary column.

4. Add a full-text string to your search to narrow the scope of your query, and then click Run query:

   service = "gateway" AND severity = "ERROR" AND "failed to query user by token"

   The results include 5,400 logs, which is fewer, but still too many.

   In the sidebar, you notice that 75% of the results are for the production-east Kubernetes cluster.

5. In the sidebar, click production-east and then click Show matching logs. Observability Platform adds the selected key/value to your query:

   service = "gateway" AND severity = "ERROR" AND "failed to query user by token"
   AND kubernetes.cluster = "production-east"

   The results include 4,000 logs. To reduce scope, begin drilling in to individual logs.

6. Expand individual logs to find commonalities across the data. You realize that the same Kubernetes pod is included in many of the logs, so you add that key/value pair to your filter:

   service = "gateway" AND severity = "ERROR" AND "failed to query user by token"
   AND kubernetes.cluster = "production-east"
   AND kubernetes.pod_name = "gateway-6agg9df321-o89ef"

   The results include less than 700 logs, which is 8,000 fewer than your initial query.

You identified the individual Kubernetes pod containing the majority of errors for the gateway service so you can inform your team and begin fixing the issue.

Access recent and saved queries

When investigating issues, you might use the same query frequently. Rather than redefining the query, use recent and saved queries to access previously defined queries in Observability Platform. You can apply a fully defined query from a previous time period by clicking a query from the Examples tab.

Recent query are available globally to all users in Observability Platform and persist for 14 days.

To access recent and saved queries:

1. In the navigation menu select Explorers > Logs Explorer (Preview).
2. Click View queries to display all available queries.
3. Click the Recent tab or the Saved tab to display the queries you want to view.
4. Locate the query you want to apply and click it.

The parameters in the query override any parameters in the query box.

Save a query

You can save queries that you access frequently so they're always available in Observability Platform. Saved queries are like bookmarks you can reference when you need them.

To save a query:

1. In the navigation menu select Explorers > Logs Explorer (Preview).
2. In the query box, construct your query. The Save query button is unavailable until you run your query.
3. Click Run query to run your query.
4. Click Save query to save your query.
5. In the Queries window, enter a name for your query and click Save.

Your query displays in the Saved tab of the Queries window. You can access your saved queries and apply them at any time.

Share a URL to a query

When investigating issues, you might want to share a defined query with other users, or include a URL to a defined query in monitor annotations, runbooks, or other on-call tools. Logs Explorer lets you copy a short URL to a defined query using either a relative or absolute time range.

Relative time is useful for understanding the results of a query in a past period of time relative to the current time, such as in the past 30 minutes. Absolute time is better suited for comparing results across a fixed point in time, such as the results of a query from last Monday at 8:00 AM versus that same query run today.

To copy a URL to a defined query:

1. In the navigation menu select Explorers > Logs Explorer (Preview).

2. Construct a query that returns the log data you want to view.

3. Click Copy URL and choose type of time range to use for the URL:

   • Copy with absolute time range: Create a link that runs a query against the date and time interval from when you copied the link. For example, if your query uses 1h as the time interval, the time range is exactly one hour ago, based on the date and time you ran the query.

   • Copy with relative time range: Create a link that runs a query against the current time. For example, if your query uses 1h as the time interval, the time range is exactly one hour ago from the current time.

The URL is copied to your clipboard based on your selection.

Logging query syntax

Use the logging query language within Logs Explorer to construct search queries for retrieving, processing, and analyzing your log data.

Observability Platform provides a lightweight, flexible syntax for querying log data in Logs Explorer. This syntax implements predefined keys that accept a comparison operator, such as an equals sign =, and a value.

KEY =|!=|=~|!~|: VALUE AND|OR (KEY =|!=|=~|!~|: VALUE) AND|OR "full text search value"
AND|OR KEY EXISTS

The query language supports full-text search by including search strings wrapped in single or double quotes, such as “36z9aaebqcc4oosuwr3v97qrgp” to find logs that contain the expression anywhere in the log. Longer, more specific full-text searches are more optimized and return results faster.

The following query matches any service named gateway where the kubernetes.namespace_name is test-logging or kubernetes.cluster_name is test and the log message contains “insert success”:

service = "gateway" AND (kubernetes.namespace_name = "test-logging" OR
kubernetes.cluster_name = "test") AND "insert success"

Features

The query syntax for native logging supports the following features:

• Autocomplete: Start typing a key to get autocomplete values. Press Control+Space to display suggestions.
• Run shortcut: Run a query by pressing Windows+Enter (Command+Return on macOS).
• Nested queries: Use parentheses () to establish the order of operations for complex queries.
• Field search: Query on specific fields by entering a key/value pair such as key = "value".
• String identification: Use either single '' or double "" quotes to identify strings.
• Full text search: Express a single value expression, such as “36z9aaebqcc4oosuwr3v97qrgp”, to find logs that contains the expression anywhere in the log.
• Exists: Find logs with a specific key by entering key.label EXISTS to return any logs containing that key and label combination. For example, kubernetes.namespace_name EXISTS returns any logs containing kubernetes.namespace_name.

Keys

The query syntax supports the following keys:

• service: Services that Observability Platform discovers in your log data. Chronosphere recommends always including a filter for service in your query for optimal performance.
• severity: Severity of issues, sanitized to the following values: DEBUG, INFO, WARN, ERROR, FATAL.
• message: Human-readable description of the log, derived from your log data.
• Custom labels you created for your logs.

Operators

After entering an operator in the query box, you can click the operator to display a dropdown with additional supported operators. This feature lets you quickly change operators in your existing query. If you modify operators with this feature, you must click Run query to submit the modified query.

The querying syntax supports the following operators:

Queries with the AND operator take precedence. If your query doesn't use parentheses, Observability Platform evaluates all AND statements sequentially, followed by any OR statements and the next set of AND queries.

The operators AND plus OR are case insensitive, so you can use AND, and, OR, and or interchangeably.

Query examples

This syntax supports nesting using parentheses so you can create complex queries. For example, the following query matches on two severity values, or where the message contains “database connection” and the mysql.table_name label starts with production.

service = "gateway" AND severity =~ "WARN|ERROR" OR (message : "database connection"
AND mysql.table_name =~ "^production.*")

The following query matches on a service named gateway with a UUID of“36z9aaebqcc4oosuwr3v97qrgp” anywhere in the log payload.

service = "gateway" AND "36z9aaebqcc4oosuwr3v97qrgp"

The following query returns all logs for a service named gateway that have a logger other than deleter, or have no logger set.

SERVICE = "gateway" AND NOT logger = "deleter"

Transformation functions

Transformation queries include functions that modify input records by adding, removing, or updating fields in the results set. Observability Platform supports the following transformation functions you can include in log queries.

timeChart()

The timeChart function shapes logs to include in a chart with a time X-axis. Use this function to visualize log queries containing time series data. The function sorts multiple time series in descending order by the last data point value.

Syntax

timeChart(function|limit|by|step)

Arguments

Example

key1 = 'val1' | timeChart(function = sum(field='price'), limit = 100, by = ['severity','service'], step = '1m')

groupBy()

The groupBy function collects logs with identical by columns. The function sorts grouped data in descending order.

Syntax

groupBy(by|function|limit)

Arguments

Example

key1 = 'val1' | groupBy(function = sum(field='production'), limit = 100, by = ['severity','service'])

tail()

The tail function returns the most recent log events.

Syntax

tail(limit)

Arguments

Example

key1 = 'val1' | tail(limit=500)

Aggregation functions

Aggregation queries include functions that can combine their input into a new structure, or emit new events into the output stream. You can use all aggregate functions in groupBy() and timeChart() functions. Observability Platform supports the following aggregation functions you can include in log queries.

count()

The count function returns a count of the number of events passing through the function.

Syntax

count()

Arguments

None.

Example

key1 = 'val1' | timeChart(function = count(), limit = 100, by = ['severity'])
key1 = 'val1' | groupBy(function = count(), limit = 100, by = ['severity'])

distinct_count()

The distinct_count function enumerates the number of distinct values of field in events passing through the function.

Syntax

distinct_count(field)

Arguments

Example

key1 = 'val1' | timeChart(function = distinct_count(hostname), limit = 100, by = ['severity'])

sum()

The sum function computes the sum of values of field over events passing through the function.

Syntax

sum(field)

Arguments

Example

key1 = 'val1' | groupBy(function = sum(field='retrycount'), limit = 100, by = ['severity'])

avg()

The avg function computes the average of values of field over events passing through the function. This function only works with fields containing numeric values.

Syntax

avg(field)

Arguments

Example

key1 = 'val1' | groupBy(function = avg(field='retrycount'), limit = 100, by = ['severity'])

percentile()

The percentile function returns the specified percentile value of a field. This function only works with fields containing numeric values.

Syntax

percentile(field|percentile)

Arguments

Example

key1 = 'val1' | timeChart(function = percentile(field='duration', percentile=95), by = ['severity'])

persisted_bytes()

The persisted_bytes function returns the sum of raw bytes.

Syntax

persisted_bytes()