Using PromQL in Observability Platform
Prometheus Query Language (PromQL) lets you select and aggregate time series data in real time.
This documentation provides an introduction to PromQL and the features and syntax most relevant to Chronosphere Observability Platform. For full details about PromQL, read the Prometheus documentation (opens in a new tab).
Observability Platform is fully compatible with PromQL (opens in a new tab). If you need labels to differentiate between clusters, namespaces, and other items, the PromQL queries in existing dashboards, alerts, and recording rules need to reflect this.
All panel types except Markdown and Service topology use queries.
Queries that return an extremely large number of data points or invalid results can result in panel errors. For example, a query might return an error for exceeding server resource limits.
Observability Platform reports these errors with an icon that appears in the corner of the Preview pane of the Add panel or Edit panel interfaces, or on the panel when viewing it on the dashboard. Hold the pointer over the icon to view the error message.
Basic querying
All PromQL expressions start with a selector, which is typically the name of the metric followed by a set of labels, values, and operators. A query returns one or more time series that match the criteria. PromQL refers to these as instant vectors because they represent a set of time series where every single data point maps to a timestamp at that instant.
|<----------metric name--------->|<---labels--->|
rate(node_network_receive_bytes_total{device=~"eth1"} [5m]) * 8
|<-------------------selector------------------>| |<->|
|<-----------------------function------------------------>| ^ operator
For example, an app updates a metric to keep a total of the bytes received by a
particular node. The app names the metric node_network_receive_bytes_total
.
The following query returns all time series that match the metric name:
node_network_receive_bytes_total{}
To return only time series with a specific label and value, include them in curly
brackets ({}
) after the metric name.
The following query returns all time series that match the metric name, and which
also have a device
label whose value is equal to eth1
:
node_network_receive_bytes_total{device="eth1"}
PromQL supports dozens of arithmetic, comparison, logical, and aggregation operators. The Prometheus documentation (opens in a new tab) provides a full list of operators.
You can combine multiple labels and values with commas.
The following query returns all time series that match the metric name, and which
also have both a device
label whose value is equal to eth1
and an instance
label whose value is equal to production
:
node_network_receive_bytes_total{device="eth1", instance="production"}
PromQL supports regular expression matching and negation in label queries:
=
: Select labels equal to the provided string.!=
: Select labels not equal to the provided string.=~
: Select labels that match the provided string based on a regular expression.!~
: Select labels that don’t match the provided string based on a regular expression.
The following query returns all time series that match the metric name, and which
also have a device
label whose value matches eth
followed by any non-zero number
of other characters:
node_network_receive_bytes_total{device=~"eth.+"}
Filter values to a range
Query results often must be filtered to a narrower time span or time-range offset. PromQL refers to these as range vectors because they represent a set of time series where every timestamp maps to a range of data points from some point in the past, as determined by the query.
The following query returns all time series with an offset
of an hour (1h
)
ago that match the metric name, and which also have a device
label whose value
matches eth0
:
node_network_receive_bytes_total{device="eth0"} offset 1h
For details about the offset
modifier, see the
PromQL documentation (opens in a new tab).
You can use PromQL to return a range vector, then use a function to perform calculations on the resulting range of time series.
The following query returns all time series from the last ten minutes that match
the metric name, and which also have a device
label whose value matches eth0
:
node_network_receive_bytes_total{device="eth0"}[10m]
Other querying features
Observability Platform also provides querying features beyond those covered by using query languages in its user interface.
- Delta queries: Query metrics that employ delta temporality, as opposed to cumulative temporality.
- Prometheus API access: Interact directly with Prometheus API endpoints for programmatic workflows.