=, and a value.
"") to find logs
that contain the filter expression anywhere in the log. Longer, more specific
full-text searches are more optimized and return results faster.
To understand which values the query syntax supports for KEY, see Keys.
Features
The query syntax for Logs Explorer supports the following features:- Autocomplete: Start typing a key to get autocomplete values. Press
Control+Spaceto display suggestions. - Run shortcut: Run a query by pressing
Control+Enter(Command+Returnon macOS). - Nested queries: Use parentheses
()to establish the order of operations for complex queries. - Field search: Query on specific fields by entering a key-value pair such as
key = "value". - String identification: Use either single
''or double""quotes to identify strings. - Full-text search: Express a single value filter expression, such as
"query user token", to find logs that contains the filter expression anywhere in the log. Full-text searches must be surrounded by double quotes. - Exists: Find logs with a specific key by entering
key.label EXISTSto return any logs containing that key and label combination. For example,kubernetes.namespace_name EXISTSreturns any logs containingkubernetes.namespace_name. See logical operators for more information.
Keys
Logs Explorer supports querying all user-defined keys in your data, in addition to the following derived keys:service: Services that Observability Platform discovers in your log data.severity: Severity of issues, sanitized to the following values:DEBUG,INFO,WARN,ERROR,FATAL.message: Human-readable description of the log, derived from your log data.- Custom labels you created for your logs.
error: code value, a valid query is structured like this example:
service as a primary key by default. Primary keys map to data
constructs in your business. For example, you might use company_id, service_name,
or event as ways to group data that’s important to your organization.
Rather than use the derived keys included in the previous list, you can define custom
primary keys, which users can then filter on Logs Explorer when searching for log
data. Chronosphere recommends always including a filter with a primary key in your
query for optimal performance.
To map your primary keys, contact Chronosphere Support.
Operators
The querying syntax for Logs Explorer supports the following operators:Logical operators
The querying syntax for Logs Explorer supports the following logical operators:
Queries with the
AND operator take precedence. If your query doesn’t use
parentheses, Observability Platform evaluates all AND statements sequentially, followed by
any OR statements and the next set of AND queries.
To use AND plus OR operators in the same query, separate them with parentheses.
For example, the following query matches any service named gateway where the
kubernetes.namespace_name is test-logging or kubernetes.cluster_name is test
and the log message contains "insert success":
AND plus OR aren’t case sensitive, so you can use AND, and,
OR, and or interchangeably.
The equal sign (=) and colon (:) operators can’t be used interchangeably. The
colon operator matches on a substring or indicates that a field contains a value. For
example, following query matches any logs where the kubernetes.cluster_name field
contains only production-us-east:
httpRequest.requestUrl key:
<, <=, >, and >= evaluate numeric field values. For
example, the following query matches logs for the nginx service where the
httpRequest.responseSize field is greater than 2000:
Mathematical operators
You can also use mathematical operators for addition (+), subtraction (-),
multiplication (*), and division (/) to complete calculations within a query.
The querying syntax supports the following mathematical operators:
This capability is especially helpful when creating monitors
to generate alerts and receive notifications. For example, the following query
includes a calculation for an alert that returns the ratio of logs without errors to
total logs in the
nginx service:
emissary-ingress service where the duration is greater than 10:
bytes_received / duration:
Transformation operators
Transformation queries include operators that reshape the result set by adding, removing, or updating fields, filtering rows, sorting, or limiting the number of returned records. Observability Platform supports the following transformation operators you can include in log queries.All transformation operators require a pipe (
|) character in a query.extend
The extend operator takes the results of a calculation and adds those columns to a
result set. Use this operator with the project operator to specify the
columns that display in the results. You can also use this operator with the
make-series, summarize, and
top-nested operators.
Syntax
Example
The following example returns logs for the
logging_service, and specifies
max_threads greater than zero to exclude rows with zero or negative values for that
field.
The first calculation creates a new column named ratio, which always returns a
value of 1 for non-zero values because it divides max_threads by itself.
The second calculation creates a new column named v2, which takes the results from
the ratio column, and then adds them to the value of max_threads (which is 1).
The project operator creates three columns in the result set, named service,
ratio, and v2, and outputs the results of the calculations to those columns.
extract
The extract operator extracts a specific capture group from a string to match on a
field or expression, through the use of a regular expression.
Syntax
Examples
The following example matches a string that includes
"error INTEGER", where
INTEGER is a numeric error code such as 503. For example, the string
"error 503" returns only the numeric error code 503 in a column named
error_code.
filter
The filter operator applies conditions to a result set after a pipe (|).
The initial filter expression at the start of a query doesn’t use a keyword. The
filter operator uses an explicit filter keyword to refine results anywhere in a
query, including after operators such as extend,
parse, join, and lookup.
The filter operator supports the same logical operators and
regular expressions as the initial filter expression. To
apply several conditions, chain multiple filter operators in a single query.
Observability Platform combines chained filter operators with AND.
The filter operator can’t filter output from the following transformation
operators:
make-seriesprojectsummarizetop-nested
Examples
The following example returns logs for the
nginx service, and then filters the
result set to logs with a severity of ERROR:
filter operator works after other operators, so it can refine a result set
following a transformation. The following example adds a response_size column
with the extend operator, filters to rows where response_size is
greater than 2000, and then projects the columns to display:
filter operators. The following example
returns logs for the checkout-service, and then narrows the result set to error
logs that contain "timeout":
filter operator is also valid inside a join subquery, where it
refines the rows that merge from the right table.
join
The join operator merges the rows of two tables into a new table by matching values
from specific columns in each table. The operator accepts a left query with a right
sub-query.
The join type can be one of the following types, and both options join all columns from
both tables, including the matching keys, but differ on the included rows:
inner: Includes only matching rows from both tables.leftouter: Includes records from the left table, and only matching rows from the right table.
==).
To ensure performant queries, you must include a primary key in the query for the
join operator. For most tenants, service is the primary key. For example:
Examples
The following example completes an inner join on logs from the
nginx service,
joining logs where severity="ERROR" on the requestID column:
"request finished", and
join the output_deployment_id with response status codes to create a table with a
column named response_class that includes the response code type:
limit
To return a specific number of results within a query, use the limit operator. You
can append this operator to any transformation operator,
such as join, make-series, summarize, and sort.
Syntax
Examples
lookup
The lookup operator reads log data from
lookup tables, and performs joins to
return data in log filters. You can combine this operator with other
transformation operators to visualize external data.
Syntax
Examples
The following example maps severity levels to descriptions and priority. The filter
completes a left outer join on the
severity_definitions lookup table, using the
severity column as reference. It then uses the project operator to
visualize several fields in the generated results table.
kubernetes_services lookup table, and
references the logging.apis.com/labels.service field against the service column
in the table. It then uses the project operator to visualize several
fields in the generated results table.
pod_ips lookup
table, and references the kubernetes.pod_ip field against the pod_ip column in
the lookup table. It then uses the project operator to visualize
several fields in the generated results table.
lookup function with the project-rename function, which
renames columns in the lookup table before completing the join operation. The
project-rename function only works with the lookup function.
This usage helps to avoid column name conflicts between log fields and values in the
lookup table, and lets you control which columns are exposed in the result set. For
example, the following filter uses the project-rename function to rename
lookup_company_name to company_name to avoid naming conflicts between the log
fields and values in the lookup table.
project-rename function to define and expose the column to the query. This
combination of functions ensures that the values in the you looked up are available
for aggregation. For example, the project-rename function in the following query
exposes the input_price_per_m column to make it available to any aggregation
functions:
make-series
The make-series operator shapes logs to include in a time chart with an X-axis. Use
this operator to visualize log queries containing time series data. You can apply
aggregations, and use step to define the time bucket size. The operator sorts
multiple time series in descending order by the last data point value.
Syntax
Examples
parse | parse-where
The parse operator accepts a column name, strings (with optional regular
expressions), and variable names to extract information out of your existing data.
Use this operator to extract parts of a field at query time that you couldn’t
otherwise use, such as a response code that’s part of a message field. The parse
operator only extracts data, while the parse-where operator filters the result set
to log data that successfully matches the parsing clause.
After extracting data, you can specify a name for the new field and also complete
aggregation calculations using any of the supported
aggregation functions. Use this function with supported
transformation operators, such as make-series,
project, summarize, and top-nested, to create a visualization from your
results.
Syntax
Examples
Consider log data that contains an improperly extracted
message field. This field
might include information like severity, environment, response time, and other useful
data, but it’s coded as a single string. You could use the substring
function to extract data, but that function relies on data being in a position that
doesn’t change.
The parse and parse-where operators let you specify an expression without relying
on position. These operators use regular expression matching by default, which you
can explicitly specify by including kind = regex in a query. This capability means
you can analyze your log data, define a query to extract data based on the structure
(rather than position), and then complete aggregations and transformations.
Assuming the overloaded message field contains a response time that’s coded as
"responseTime": VALUE, the log data looks similar to this:
severity could be several different values (such as
INFO, WARN, or CRIT), so you can’t guarantee the position of the response time
value, meaning you can’t use the substring() function. Instead, use the parse
operator to extract that value from your log data and then compute the average:
- Matches on logs that contain
"responseTime". Use single quotes ('') to escape double quotation marks. - Specifies a regular expression to match any data starting with
"responseTime":and ending with a comma (,). - Names the computed data
time. - Takes the average of
timeand returns the data in a table.
make-series operator:
"responseTime", use the parse-where
operator instead of parse:
message field with varied information:
parse clauses in a
single query:
- Matches all logs for the
zookeeperservice. - Extracts part of a string starting with
Processingand ending withcommand, and names the new fieldcommand. - Extracts part of a string after
127.0.0.1:, and names the new fieldport. - Projects the output of the
commandandportfields in a table.
project
The project operator selects columns to include or compute in a logging query. Use
this operator to return data in a table with the specified fields as table columns.
If you want to return data in a tabular format and also complete aggregations and
sorting, use the summarize operator instead.
You can also use the project() operator with the substring function
to extract a substring from the source string based on a starting index character
position. For example,
project() operator to select fields and customize columns and rows before
downloading log data.
Syntax
Examples
sort
The sort operator sorts your results by a specified column, in either ascending or
descending order. The operator supports sorting by multiple columns, such as by both
service and severity. When sorting by multiple columns, you can specify the sort
order for each column.
When sorting results that include an aggregation query, use
an alias to sort by instead of the aggregation field name.
Sorting by the alias provides better control and predictability in your queries.
For example, the following query retrieves all logs that have a severity of
WARNING, aggregates the results in a table, and returns the sum of the
httpRequest.responseSize field, ordered by service:
total, which simplifies the subsequent sort by
clause:
Examples
summarize
The summarize operator generates a table of data that aggregates the content of the
table from the input query, grouped in descending order. By default, this operator
uses the count() function to count all permutations of the specified
field if no AGGREGATION is specified.
Input rows are arranged into groups that have the same expression. Use the by
keyword to specify the columns you want to group your data by before applying any
aggregations.
Your query can include only one summarize operator. To group data and refine the
results with subsequent clauses, use the top-nested operator.
Syntax
Examples
top-nested
The top-nested operator completes aggregation and value selection in a hierarchical
manner. The operator partitions data based on criteria from the first top-nested
clause, and then selects the top records in each partition using an aggregation, if
specified.
Unlike the summarize operator, you can include multiple top-nested
clauses in a single query. Using a single pipe (|) separator, specify a “root”
top-nested clause, and then nest additional top-nested clauses separated by a
comma. Each of these clauses refine the partitions from the previous clause, creating
a hierarchy of more precise groupings.
For example, using the summarize operator, the following query returns the count of
logs from each endpoint from the nginx-requests service in descending order:
summarize operator can use additional aggregation functions, but on its own
returns a two-column table containing a count of the specified field. The
top-nested operator is similar, but provides additional options to partition and
display results.
The top-nested operator creates a table with two columns for each clause. One
column contains unique values from the filter EXPRESSION, and the other column
shows the results obtained from the AGGREGATION calculation.
Using the top-nested operator, the following query refines the results from the
nginx-requests service to the top 80 endpoints, and further partitions the data by
the top nine HTTP response status codes:
- The name of the endpoint from
http.response.endpoints. - The aggregated value for the endpoint.
- The HTTP status code from
http.response.status_code. - The aggregated status code for the endpoint.
http.response.endpoints times the number of values returned for
http.response.status_code.
Syntax
Examples
Query refinements
Logs Explorer provides query refinements such as aliasing field names, matching with regular expressions, and querying array values.Alias field names
When running a query, you can add an alias for a field name that appears in generated visualizations, such as in dashboards and in signal groups for alerts. This capability is similar to creating an alias for function names. Themake-series, project, and
summarize operators support field aliasing.
For example, the following query creates a time chart that uses k8 as an alias for
the k8s.namespace field. The generated chart uses k8 instead of the full field
name.
total for the count()
function, and svc for the service field. The generated chart includes results
such as svc:nginx total. Without the aliases, the same result would be
service:nginx count_.
k8s.namespace. Fields containing periods are converted to underscores in monitors
that use signals, so k8s.namespace
displays as k8s_namespace in the monitor query visualization.
Regular expressions
Logs Explorer uses re2 regular expression syntax. The following regular expressions are supported, and can be used between fields and values:Queries that use regular expression operators match only on the first 1,024
characters of a string. These operators won’t match any characters in a string that
exceed 1,024 characters. To change this match limit, contact
Chronosphere Support.
kubernetes.cluster_name
contains the following values: production-1, production-2, production-3,
productionNEW, old_production.
The following query matches the word production anywhere in the
kubernetes.cluster_name field, so it matches all values in the field:
.*), which
matches the word production, followed by zero or more characters anywhere in the
phrase. This query also matches all values in the field:
-*) matches the word production, followed by zero or
more dash characters anywhere in the phrase. This query also matches all values in
the field, including productionNEW and old_production, which might not be
immediately apparent:
production-, combine the dash and period
characters with a wildcard and add a dollar sign ($). The following query matches
only production-1, production-2, production-3, but doesn’t match either
productionNEW or old_production:
Arrays
You can query array values up to the first level. During ingestion Observability Platform flattens multiple level arrays to one level to preserve order. If a key precedes the array index, the index must always be at the end. For example, consider the following array of objects belonging to thenginx service
as they display in Logs Explorer:
1) that match events = e, use the
following query:
Query examples
This syntax supports nesting using parentheses so you can create complex queries. For example, the following query matches on two severity values, or where the message contains"database connection" and the mysql.table_name label starts with
production:
gateway where the kubernetes.pod_id
equals 3bf26945-b817-4e72-b22c-662c318af2f1 anywhere in the log payload:
gateway that have a
logger other than deleter, or have no logger set:
Aggregation functions
Aggregation queries include functions that can combine their input into a new structure, or emit new events into the output stream. You can use all aggregation functions withsummarize and make-series operators,
with the exception of arg_max(), which can be used only with the
summarize operator.
Observability Platform supports the following aggregation functions you can include
in log queries.
Group results
To group results in queries that include an aggregation function, use theby
operator. This operator lets you group results by a specified field, which is
different than the sort operator, which lets you sort results by a
specified column.
For example, the following query includes the summarize
transformation operator to return the average for the httpRequest.responseSize
field, and then groups the results by service:
Alias function names
When running a query containing a transformation operator and an aggregation function, you can add an alias for the function name that displays in the generated visualization. This capability applies to all transformation operators and aggregation functions. For example, the following query creates a table of data that aggregates the content of the table from the input query, and displays the count byservice. The column
header that contains the total count is named _count.
total for the
count() function. The column header containing the total count is named total.
make-series operator:
arg_max()
The arg_max function returns the value of an expression for the row with the
largest _timestamp. The expression determines which column values to return, from
the row with the largest _timestamp values.
By default, the function creates a row header named arg_max_EXPRESSION, where
_EXPRESSION_ is the expression included as the argument. For example, if your
function call is arg_max(service), the resulting column name is arg_max_service.
To specify a more human-readable and intuitive column name, use
an alias.
Syntax
Examples
The following example returns values for
service with the largest _timestamp,
ordered by severity:
summarize
operator with the arg_max function. The example also uses an alias named
service_with_largest_message, which becomes the column header in the generated
table.
array_length()
The array_length function returns the length of an array field. The function takes
a field name as input and returns a numeric value representing the number of elements
in the array. This function works only with fields containing numeric values.
Syntax
Examples
The following example filters logs to the
nginx service and completes the following
transformations:
- Adds an
upstream_lengthcolumn with the array length of theupstreamAddrfield. - Selects only the
service,upstreamAddr, andupstream_lengthcolumns. - Sorts the results by the
upstream_lengthcolumn in descending order.
avg()
The avg function computes the average of values of a filter EXPRESSION over
events passing through the function. This function only works with fields containing
numeric values.
Syntax
Examples
avgif()
The avgif function computes the average of values of a field where the specified
filter EXPRESSION evaluates to true. This function works only with fields
containing numeric values.
Syntax
Examples
case()
The case function accepts a conditional expression that evaluates conditions in
order, and returns the value of the first matching condition. If none of the
conditions evaluate to true, the result of the else expression is returned.
Specify one or more condition-value pairs as condition, value in the filter. Use
this function in conjunction with the
transformation operators to specify conditional
expressions.
Syntax
Examples
This example filters logs for the
nginx service in all environments starting with
prod, where the number of bytes sent is less than or equal to 50000. The filter
uses the case function to categorizes bytes sent based on these conditions: small
(≤1000), medium (≤5000), and large (>5000).
The project operator creates three columns named buckets, kubernetes.pod_name,
and environment, and then sorts the results by kubernetes.pod_name.
count()
The count function returns a count of the number of events passing through the
function.
Syntax
Examples
countif()
The countif function returns a count of rows where the specified filter
EXPRESSION evaluates to true. This function ignores null values.
Syntax
Examples
count_distinct()
The count_distinct function returns the exact count of distinct values for a field.
Use this function when you need exact results. To return approximate results, use the
dcount function, which is more performant and provides suitable accuracy
for most use cases.
Syntax
Examples
count_distinctif()
The count_distinctif function combines the count_distinct
function with a filter to return the exact count of distinct values for a field,
where a specified condition is true. To return approximate results where a condition
is true, use the dcountif function, which is more performant and
provides suitable accuracy for most use cases.
Syntax
Examples
dcount()
The dcount function calculates an estimated number of distinct values of a FIELD
in events passing through the function.
Chronosphere recommends using this function in almost all scenarios.
If you need exact results, use the count_distinct function.
Syntax
Examples
dcountif()
The dcountif function calculates an estimated number of distinct values where the
specified EXPRESSION filter evaluates to true. Use this function to estimate the
cardinality of large data sets.
The
dcountif function trades accuracy for performance, and might return a result
that varies between executions.
Examples
floor()
The floor function rounds a numeric value down to the nearest integer, and returns
a dynamic array of the calculation.
Syntax
Examples
The following query converts response sizes from bytes to kilobytes, and then returns
the floor of the result using the
floor() function. The query includes the
extend() function to create a new column named rounded_response_size
for the results of the floor calculation.
hash()
The hash function takes an input string, uses the xxHash 64
algorithm to hash it, and returns the hash as a 64-bit integer. Use this function in
conjunction with the
replace fields
control rule to hash values, such as shortening long, repetitive strings in your
data. Hashing values at ingest time avoids storing the entire field, which decreases
storage costs. You can then use the hash function to query the hashed fields.
Syntax
Examples
Use the
hash function to query hashed values, such as ones that replace long error
messages or stack traces. For example, return all error messages and hash the value
of the message field, and sort by count_ in descending order.
extend
and project. For example, use a hashed value in place or the HTTP
request from a service to shorten the value and improve query performance.
min()
The min function returns the minimum value of the specified field.
Syntax
Examples
minif()
The min function returns an estimate of the minimum value of the specified field
where the EXPRESSION filter evaluates to true.
Syntax
Examples
max()
The max function returns the maximum value of the specified field.
Syntax
Examples
maxif()
The maxif function returns an estimate of the maximum value of the specified field
where the EXPRESSION filter evaluates to true.
Syntax
Examples
moving_average()
The moving_average function accepts an AGGREGATION containing a dynamic numeric
array as input, and applies a
simple moving average (SMA)
filter.
This function requires an aggregation with an expression that operates on a specified
time interval to calculate a moving average for. The combination of the AGGREGATION
and EXPRESSION defines the calculation for which this function operates.
Syntax
Examples
The following example calculates a series of summations for
request_size over the
last minute (1m), takes a moving average of the last five (5) summations, and
then divides by 1000 to convert the results to megabytes. The results are ordered
in a time chart with an X-axis, sorted by severity.
percentile()
The percentile function returns the specified percentile value of a filter
EXPRESSION. This function only works with fields containing numeric values.
Syntax
Examples
replace_regex()
The replace_regex function replaces strings based on the results of a regular expression. The
function finds all matches on an input string using a regular expression pattern, and
replaces them with a specified string.
Use the replace_regex function to clean, normalize, and transform string fields in
log queries. Replacements can use fields with a simple regular expression, or use
patterns to match on data.
Syntax
Examples
The following query uses a simple replacement that replaces all instances of
[warn]
with error in the message field for the nginx service:
nginx service with a severity of "ERROR", where the message field contains a
string like:
replace_regex function with a capture group ([^.\s]+) to match
the hostname that precedes example.com, replaces the entire message with the string
matching the regular expression, and projects the original message and the
extracted hostname.
round()
The round function returns a numeric value that is closest to the nearest integer, with an option
to specify decimal precision. If no decimal precision is specified, this function
rounds to the nearest integer.
Although the floor function always rounds down to the nearest integer, the
round function can round values up or down, depending on the nearest integer value.
Syntax
Examples
Use this function with a field or a math expression. For example,
round(duration)
rounds the value of the duration field to the nearest integer. If you specify a
math expression such as round(1 + 3, 1), the function evaluates the math expression
first, and then rounds the result to one decimal place.
The following filter uses the round function to convert response sizes from bytes
to kilobytes (dividing by 1,000), and then rounds to two decimal places. The
extend function creates a new column named rounded_response_size, and the
project function selects the original response size and the
rounded_response_size column for comparison.
string_size()
The string_size function returns the length of a string expression as a numeric
value, which represents the number of characters in the string.
Syntax
Examples
The following example uses the
extend operator to create a column named
size, and use string_size to calculate the character count of the message
field.
extend operator to create a column named
msg_size, and use string_size to calculate the character count of the message
field. The filter also uses the summarize operator to compute the
average message size per service.
substring()
The substring function extracts a substring from the source string based on a
starting index character position.
Syntax
Examples
The following query includes a colon (
:) to find all logs that have a
resource.type field containing a k8s substring. Then, the query groups the logs
by the resource.type field, groups the logs by the first 13 characters of the k8s
substring, and returns those groups.
resource.type shows that this query matches values such
as k8s_container, k8s_resources, and k8s_cluster.
The following query uses the substring() function as a filter without any
aggregation. This kind of query is helpful when you want to ensure that a particular
substring matches a value. In this case, the query matches any logs containing
writer in the kubernetes.pod_name field:
sum()
The sum function computes the sum of values of a filter EXPRESSION over events
passing through the function.
Syntax
Examples
sumif()
The sumif function computes the sum of values for the specified field where the
EXPRESSION filter evaluates to true.
Syntax
Examples
sum(_payloadSize)
The sum(_payloadSize) function calculates the size of the log payload in bytes.
Syntax