Rollup rules

To downsample and aggregate metrics after they're sent by the client but before they're stored, create rollup rules.

Rollup rules are a type of aggregation rule that help you reduce the cardinality footprint of your metrics by dropping raw data to eliminate unneeded labels. High cardinality footprints can cause slow dashboards and queries.

If you're working with late-arriving data, rollup rules are well suited for ensuring all of your data aggregates the way you need it.

As an example, instance or pod labels don't often add value on their own, but removing these labels from the client side isn't always possible. You can use rollup rules to avoid storing these labels.

Rollup rules support both Prometheus and Graphite metrics.

Rollup rule attributes

To accurately aggregate your data, rollup rules require you to both configure multiple fields and to have an understanding of aggregation operations.

Fields for rollup rules

Each rollup rule must declare the type of metric it aggregates by setting the metric_type field, because each metric type aggregates differently. Choosing the wrong metric_type for your rule can produce unexpected results.

• aggregation: Specifies how to combine the grouped metrics. See the supported aggregation operations for specific help.

• metric_type: The metric type aggregated. Valid options vary depending on your data. See the supported aggregation operations for specific help.

• filter: Filters incoming metrics by label. If multiple label filters are specified, an incoming metric must match every label filter to match the rule. Label values support glob patterns, including matching multiple patterns with an OR, such as service:{svc1,svc2}.

  Several special filters are available for matching metrics by non-label request metadata:

  Filter	Description	Valid values
  __metric_type__	Matches on the incoming metric's Chronosphere metric type. This is the recommended method for filtering on metric type.	cumulative_counter, delta_counter, gauge, or measurement
  __m3_prom_type__	When ingesting with Prometheus, matches on the incoming metric's Prometheus metric type.	counter, gauge, histogram, gauge_histogram, summary, info, state_set, or quantile
  __otel_type__	When ingesting with OpenTelemetry, matches on the incoming metric's OpenTelemetry metric type.	sum, monotonic_sum, gauge, histogram, exp_histogram, or summary
  __otel_temporality__	When ingesting with OpenTelemetry, matches on the incoming metric's OpenTelemetry temporality.	delta or cumulative
  __m3_type__	DEPRECATED. Matches on the incoming metric's legacy M3 type.	counter, gauge, or timer

  Example:

  __metric_type__:cumulative_counter service:gateway __name__:http_requests_*

  This filter matches any cumulative counter metric with a service=gateway label whose metric name starts with http_requests_.

• A label policy: Label policies act as a filter, defining which labels to preserve in the resulting metric. Use group_by to keep one or more labels, or exclude_by to ignore one or more labels. (Known as keep and discard in the Aggregation Rules UI).

• new_metric: Optional for Graphite. The name of the new metric to create and persist to the database. You can use the template string {{ .MetricName }} in the new_metric to create a new metric name that references the original metric name. For instance, new_metric: '{{ .MetricName }}:by_instance' outputs a metric with the name my_metric:by_instance if the matched metric is my_metric.

• interval: The distance in time between aggregated data points. Intervals are based on your retention policy. Use this optional field to set a custom interval. (Known as storage_policies in version 0.286.0-2023-01-06-release.1 and earlier.)

• drop_raw: Defaults to false. Set to true to remove raw metrics that match this rollup rule. For more information, see Mapping rules.

Label policies

You define which labels to preserve in the resulting metric through the use of label policies. To do this, add the appropriate field to the rollup rule definition:

Only one of group_by or exclude_by can be set per rollup rule.

Graphite metrics support only the exclude_by rule type.

• group_by (keep in the Aggregation Rules UI)

  When using group_by rollup rules, you must specify the labels by which to aggregate the metrics. The rule aggregates only metrics that contain all of the keep labels. group_by retains only the selected labels and discards any other labels. If a metric doesn't include all of the labels specified by group_by, the metric isn't included in the rule.

  Use a group_by rule when there are individual metrics you can filter with the __name__ label that you want to aggregate.

• exclude_by (discard in the Aggregation Rules UI)

  With an exclude_by rollup rule, you specify which labels to remove from the aggregated metric, while keeping all other labels.

  Use an exclude_by rule when you want to target a group of metrics for a particular service, team, or other higher level set of metrics.

Supported aggregation operations

Some operations can change the type of the metric during aggregation. The resulting metric type of an aggregation is called the output metric type.

Even if you are ingesting data with the wrong metric type, configure your rollup rule with the metric type that the ingested data should be. For example, if Chronosphere ingests metrics with type GAUGE, but the values actually represent DELTA_COUNTER, use a metric_type=DELTA_COUNTER rollup rule to aggregate them.

Rollup rules support the following aggregation operations:

CUMULATIVE_COUNTER

Also known as COUNTER.

Cumulative counters support these aggregations:

• SUM: Takes the increase of each individual input series within the configured interval, then sums the increases together according to the configured label policy. The output is the cumulative summed increase across all input series.

• COUNT: Counts the number of unique input series matched by the configured label policy (for example, cardinality).

The output type of all cumulative counter aggregations is a CUMULATIVE_COUNTER.

GAUGE

Gauges support the following aggregation methods:

• SUM: Takes the max value of each individual input series within the configured interval, then sums all final values together by the configured label policy.

• COUNT: Counts the number of unique input series matched by the configured label policy (for example, cardinality).

• MIN: Takes the minimum value of all data points within the configured interval across all series matched by the configured label policy.

• MAX: Takes the maximum value of all data points within the configured interval across all series matched by the configured label policy.

• PXX, MEAN, MEDIAN, STDEV, SUMSQ: Takes the maximum value of each individual input series within the configured interval, then computes the desired value distribution.

The output type of all gauge aggregations is a GAUGE.

When querying a gauge metric with a range vector included in the query downsampling might impact the accuracy of the query result. Most use cases that fit this criteria can be converted to use counters instead, which avoids the issue.

DELTA_COUNTER

Also known as DELTA.

Supported aggregations:

• SUM: Sums all values of all series matched by the configured label policy. All values must be nonnegative.

• COUNT: Counts the number of unique input series matched by the configured label policy (for example, cardinality).

• COUNT_SAMPLES: Counts the number of input samples matched by the configured label policy.

The output type of all delta counter aggregations is a cumulative counter.

Exceptions for DELTA_COUNTER metrics

DELTA_COUNTER metrics don't require the following fields for rollup rules:

• name
• aggregation
• keep
• discard

MEASUREMENT

A key feature of MEASUREMENT aggregations lies in how they treat individual samples. Unlike other types such as GAUGE and CUMULATIVE_COUNTER, MEASUREMENT metrics aggregate all at once, across all samples of your matching time series within the aggregated time interval. This enables calculation of accurate statistics server-side, within Chronosphere.

A typical use case for MEASUREMENT aggregations is calculating statistics across raw request latencies across all instances. This can be correctly performed through metric_type=MEASUREMENT and aggregation=P95. Using metric_type=GAUGE in this scenario produces undesired results, discarding all samples except the per-instance max value, then computing the 95th percentile across these per-instance max values.

Measurements support all aggregation methods:

• SUM: Sums all values of all series matched by the configured label policy. All values must be nonnegative. The output metric type is a CUMULATIVE_COUNTER.

• COUNT: Counts the number of unique input series matched by the configured label policy (for example,cardinality). The output metric type is a CUMULATIVE_COUNTER.

• COUNT_SAMPLES: Counts the number of input samples matched by the configured label policy. The output metric type is a CUMULATIVE_COUNTER.

• LAST: Takes the last value of all samples matched by the configured label policy. The output metric type is a GAUGE.

• MIN: Takes the minimum value of all samples matched by the configured label policy. The output metric type is a GAUGE.

• MAX: Takes the maximum value of all samples matched by the configured label policy. The output metric type is a GAUGE.

• PXX, MEAN, MEDIAN, STDEV, SUMSQ: Computes the desired value distribution across all samples matched by the configured label policy. The output metric type is a GAUGE.

View rollup rules

In the Chronosphere app, view rollup rules in the Aggregation rules UI.

To list only rollup rules on the command line, use Chronoctl. The command chronoctl rollup-rules list returns all rollup rules.

For information about viewing, copying, or downloading rule configurations, see Rule configuration.

Create a rollup rule

You can apply rollup rules by using either Chronoctl or Terraform. Chronosphere doesn't limit the number of rollup rules a system can have.

Rollup rules take effect immediately, but can require a full recording interval to show a change.

Best practices for rule creation

Following these guidelines helps ensure your rollup rules work as intended:

• Before using a rollup rule to group labels, be sure those labels aren't used in other places, such as dashboards, monitors, or the queries you use to debug issues.
• Filters using curly braces ({}) shouldn't use a dash (-) in the filter for label names. The single filter identifies this as a range. For example, service_cluster: !{my-label} fails. Rewrite the filter to service_cluster:!human-label instead.
• Metrics can match more than one rule. Matching multiple rules can affect data retention.

Delete rollup rules

Delete rollup rules with Chronoctl with the chronoctl rollup-rules delete command. Provide the slugs of the rules to delete.

chronoctl rollup-rules delete SLUG

For example, to delete the http_request_duration_by_service_and_status rule, use this command:

chronoctl rollup-rules delete http_request_duration_by_service_and_status

If your slug starts with a dash (-), use double quotes (") around the slug name.

chronoctl rollup-rules delete "/-my-rollup-rule"