Rollup rules

To downsample and aggregate metrics after they’re sent by the client but before they’re stored, create rollup rules. Rollup rules are a type of aggregation rule that help you reduce the cardinality footprint of your metrics by dropping raw data to eliminate unneeded labels. High cardinality footprints can cause slow dashboards and queries. If you’re working with late-arriving data, rollup rules are well suited for ensuring all of your data aggregates the way you need it. As an example, instance or pod labels don’t often add value on their own, but removing these labels from the client side isn’t always possible. You can use rollup rules to avoid storing these labels. Rollup rules support both Prometheus and Graphite metrics.

View rollup rules

Select from the following methods to view rollup rules.

Web
Chronoctl
API

In Observability Platform, view rollup rules in the Aggregation rules UI.

To list only rollup rules on the command line with Chronoctl, use this command:

chronoctl rollup-rules list

To complete this action with the Chronosphere API, use the ListRollupRules endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

For information about viewing, copying, or downloading rule configurations, see Rule configuration.

Create a rollup rule

Select from the following methods to apply rollup rules. Observability Platform doesn’t limit the number of rollup rules a system can have.

If you define a rollup rule using the Observability Platform app, you must download the rule configuration and apply it with one of the supported methods.

Web
Chronoctl
Terraform
API

Create rollup rule configurations in Observability Platform from the Aggregation rules UI.When creating a rule configuration, the Visual Editor displays by default. When creating a rule in Metrics Analyzer, the dialog pre-populates fields based on the user’s selected data.To create a rule configuration:

Enter or edit data for the following fields:
- Rule Name: Add or edit the name of the rule.
- Rule Details: Either Rule Preview or Rule Enabled. See rule mode for more information.
- Matching Time Series: Time series the rule applies to. You must include a Label, operator (= or !=), and a Value. The value you enter maps to the filters section of the CreateRollupRule endpoint. For example, if you want the rollup rule to match on Prometheus gauge metrics, enter __m3_prom_type__ as the label to match on, and gauge as the value. The resulting filter looks like:
  __m3_prom_type__ = gauge
  Values accept a comma-separated list and glob syntax, including matching multiple patterns with an OR, such as service:{svc1,svc2}. Click Add to add another time series.
- Labels to Roll Up: Discard Labels or Keep Labels. Add labels to the Input Labels text box.
- Output Metric: The new metric’s name and aggregation configuration.
  - Output Metric Name: Edit the output metric name. Clear the checkbox for Include metric name to remove the original name.
  - Input Metric Type: Select a metric type, which determines how the rollup rule interprets all matching data points. For example, if you select Gauge, the rollup rule interprets all matching data points as that data type, even if the original source isn’t a gauge metric. This behavior means that the metric type you choose doesn’t have to match the data type of the incoming data. If you want to match the incoming metric to a specific type, enter two matching time series in the rollup rule: one to match the metric, and another to match the metric type. Use __metric_type__ to define the type of metric you want to match on. For example, if you want to match a time series named agg_write_latency that’s a cumulative exponential histogram, define two series that look like:
    __name__ = agg_write_latency AND __metric_type__ = cumulative_exponential_histogram
  - Aggregation: Select an aggregation operation.
  - Aggregation Interval: The length of time between samples. See Aggregation interval.
- Raw Data: Turn on Drop raw input data to remove the raw input data after aggregation.
When finished, click Code Config.
Choose your rule creation method from these options:
- Chronoctl
- Terraform
- API
Apply the changes based on your selected method.

To create a rollup rule with Chronoctl, define the rule in a YAML file and apply it.If you don’t already have a YAML configuration file, use the scaffold Chronoctl parameter to generate a template for a specific resource type:

chronoctl rollup-rules scaffold

You can redirect the results (using the redirection operator >) to a file for editing.

Create or edit a YAML configuration file to configure the rollup rule.
Apply the rollup rule:
chronoctl apply -f FILE_NAME.yaml

Replace FILE_NAME with the name of the YAML configuration file.See the Chronoctl rollup rule example for more information.

When you run terraform plan to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the Terraform provider documentation.

Create a rollup rule with Terraform by using the chronosphere_rollup_rule type, followed by a name in a resource declaration. See the Terraform rollup rule example for more information.

Add the definition to a Terraform file.
Run this command to create the resource:
terraform apply

To complete this action with the Chronosphere API, use the CreateRollupRule endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Rollup rules take effect immediately, but can require a full aggregation interval to show a change.

Best practices for rule creation

Following these guidelines helps ensure your rollup rules work as intended:

Use Live Telemetry Analyzer to verify your glob syntax to ensure your query matches the correct metrics.
Before using a rollup rule to group labels, be sure those labels aren’t used in other places, such as dashboards, monitors, or the queries you use to debug issues.
A dash (-) inside square brackets is a range operator, not a literal character. The filter service_cluster:[a-d] matches a single character from a through d. To match a value that contains a literal dash, use curly braces, which take a comma-separated list of alternatives and treat a dash as an ordinary character. For example: service_cluster:{my-label,other-label}.
Metrics can match more than one rule. Matching multiple rules can affect data retention. If a rule matches any drop_raw=true, raw metrics are dropped.
If a single output series receives more than 10 million unique input series, Observability Platform might stop accepting new input series specified in the rollup rule, which could result in partially aggregated metrics. To avoid this behavior, choose a label policy that writes more output series by removing fewer labels.

Chronoctl rollup rule example

Here’s an example of a rollup rule that matches time series with the value permits_blocked, while discarding any labels matching instance and job. It uses a cumulative counter type metric, and aggregates as a sum using a 30-second interval.

Chronoctl example

api_version: v1/config
kind: RollupRule
spec:
  slug: permits_blocked_without_instance
  name: permits blocked without instance
  filters:
    - name: __name__
      value_glob: permits_blocked
  metric_name: '{{ .MetricName }}:without_instance'
  metric_type: CUMULATIVE_COUNTER
  aggregation: SUM
  interval: 30s
  label_policy:
    discard:
    - instance
    - job
  add_metric_type_label: true
  mode: ENABLED

Terraform rollup rule example

Terraform example

resource "chronosphere_rollup_rule" "permits_blocked_without_instance" {
  name      = "permits blocked without instance"
  slug      = "permits_blocked_without_instance"
  filter    = "__name__:permits_blocked"
  metric_type = "CUMULATIVE_COUNTER"
  aggregation = "SUM"
  interval = "30s"
  exclude_by  = ["instance", "job"]
  metric_type_tag = true
  mode      = "ENABLED"
  new_metric  = "{{ .MetricName }}:without_instance"
}

Delete a rollup rule

Chronoctl
Terraform
API

To delete rollup rules with Chronoctl, use this command:

chronoctl rollup-rules delete SLUG

Replace SLUG with the rule’s slug.For example, to delete the http_request_duration_by_service_and_status rule, use this command:

chronoctl rollup-rules delete http_request_duration_by_service_and_status

If your slug starts with a dash (-), use double quotes (") around the slug name.

chronoctl rollup-rules delete "-my-rollup-rule"

To delete a resource that’s managed by Terraform:

Edit your Terraform configuration file to remove the pre-existing resource definition.
Run this command to remove the resource from Observability Platform:
terraform apply

To complete this action with the Chronosphere API, use the DeleteRollupRule endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Rollup rule attributes

To accurately aggregate your data, rollup rules require you to both configure multiple fields and to have an understanding of aggregation operations. See the CreateRollupRule API documentation for the complete list of fields that are part of the rollup_rule object that you define when creating a rollup rule with any of the supported methods.

Filters

The filters field selects the incoming metrics a rule applies to. Each entry matches one label name against a value, and a metric must match every entry to match the rule. Values support glob syntax, including matching multiple patterns with an OR, such as service:{svc1,svc2}. Synthetic labels match on request metadata instead of on a metric’s label. The following table lists the synthetic labels that a rollup rule filter most often matches on, and the values each one accepts:

Filter	Matches	Valid values
`__metric_type__`	The metric type. Use this filter to match on metric type.	`cumulative_counter`, `cumulative_exponential_histogram`, `delta_counter`, `delta_exponential_histogram`, `gauge`, `measurement`
`__metric_source__`	The source format the metric arrived in.	`carbon`, `chrono_azure`, `chrono_cloudflare`, `chrono_gcp`, `chrono_integration`, `cloudwatch_metric_stream`, `dogstatsd`, `open_metrics`, `open_telemetry`, `prometheus`, `signalfx`, `statsd`, `tagged_statsd`, `wavefront`
`__m3_prom_type__`	The Prometheus metric type, for data ingested with Prometheus.	`counter`, `gauge`, `histogram`, `gauge_histogram`, `summary`, `info`, `state_set`
`__otel_type__`	The OpenTelemetry metric type, for data ingested with OpenTelemetry.	`sum`, `monotonic_sum`, `gauge`, `histogram`, `exp_histogram`, `summary`

For example, the following filter matches any cumulative counter that carries a service=gateway label and whose metric name starts with http_requests_:

__metric_type__:cumulative_counter service:gateway __name__:http_requests_*

Aggregation interval

The interval field sets the amount of time between the aggregated data points a rule produces. If you omit it, the rule uses the default resolution from your retention policy. An interval must match a resolution that your tenant can write aggregated data to, typically 15s, 30s, or 60s.

Rule mode

The mode field controls whether a rule aggregates data. It accepts two values:

ENABLED: Aggregates incoming data according to the rule configuration. A rule that omits mode uses this value.
PREVIEW: Reports the rule’s estimated impact without changing stored data. For the preview workflow, see Preview a rule’s shaping impact.

Conflicting output metric names

Set skip_on_conflict to true to skip a rule when another rollup rule already produces a metric with the same output name. Use this field to write a rule that produces a series only when no other rule produces one under that name. Default: false.

Label policies

Use label policies to define which labels to preserve in the resulting metric. In the rollup rule definition, add the appropriate field to specify which labels to retain or discard. Most rollup rules must set one of these fields, and no rule can set both. Neither field accepts __name__. A rollup rule can’t group by the metric name or drop it. To change the output metric’s name, use metric_name instead.

Rules that set graphite_label_policy can’t set keep or discard, their Terraform equivalents group_by and exclude_by, or metric_name. Manage Graphite positional labels with graphite_label_policy.replace instead.

Keep specified labels

To aggregate only metrics that contain all of the specified labels and discard all other labels, use group_by (Terraform) or keep. When using these rollup rules, you must specify the labels to aggregate the metrics by. If a metric doesn’t include all of the specified labels, the metric isn’t included in the rule. If a rollup rule uses group_by or keep, the rule will match only metrics with labels that contain these fields, even if the label filters would have matched these metrics. A rule that uses group_by or keep also constrains its own filters. The filters must include a __name__ entry, and that entry can’t contain a wildcard, so the rule targets one metric name instead of an arbitrary set. Adding the {{ .MetricName }} template to metric_name lifts both restrictions, because the output name then varies with each matched metric.

Remove specified labels

To target a group of metrics for a particular service, team, or other higher-level set of metrics, use exclude_by (Terraform) or discard. When using these rollup rules, you specify which labels to remove from the aggregated metric, while keeping all other labels.

Set a Graphite label policy

For Graphite metrics, you can use the graphite_label_policy parameter to also set a Graphite-specific label policy. This lets you define replacements for label values without changing their positions, which can reduce cardinality without breaking Graphite metrics’ preferred positional indexing. For example, assume you have raw metric names that follow this pattern:

cluster.production.instance.instance1.requests_count
cluster.production.instance.instance2.requests_count
...

You can create a Graphite label policy that defines a replacement rule that replaces the third positional label name (__g3__) with a new string value (INSTANCE). This replacement aggregates these metrics as cluster.production.instance.INSTANCE.requests_count, without changing their positional indexing.

Chronoctl
Terraform

The output of the chronoctl rollup-rules scaffold command includes the graphite_label_policy parameter:

Chronoctl example

api_version: v1/config
kind: RollupRule
spec:
  # ...
  graphite_label_policy:
    # List of labels to replace. Use to discard high-cardinality values while still
    # preserving the original positions of the Graphite metric.
    replace:
      - # Name of the label to replace the value for. Only positional Graphite labels
        # such as '__gX__' are allowed.
        name: <string>
        # New value of the replaced label.
        new_value: <string>
  # ...

To implement the rule from the example scenario as a Chronoctl YAML resource, define the name and new_value in the list of replace values:

Chronoctl example

api_version: v1/config
kind: RollupRule
spec:
  # ...
  graphite_label_policy:
    replace:
      - name: "__g3__"
        new_value: "INSTANCE"
  # ...

Define multiple replacements in a single rollup rule by adding more pairs of name and new_value to the replace list.

The graphite_label_policy block in the chronosphere_rollup_rule resource contains a replace block with name and new_value arguments.For example, to implement the rule from the example scenario as a Terraform resource:

Terraform example

resource "chronosphere_rollup_rule" "rollup_rule" {
  # ...
  graphite_label_policy {
    replace {
      name      = "__g3__"
      new_value = "INSTANCE"
    }
  }
  # ...
}

Aggregation operations

Some operations can change the type of the metric during aggregation. The resulting metric type of an aggregation is called the output metric type. Even if you are ingesting data with the wrong metric type, configure your rollup rule with the metric type that the ingested data should be. For example, if Chronosphere Observability Platform ingests metrics with type GAUGE, but the values actually represent DELTA_COUNTER, use a metric_type=DELTA_COUNTER rollup rule to aggregate them. Rollup rules support the following aggregation operations:

`CUMULATIVE_COUNTER`

Cumulative counters support these aggregations:

SUM: Takes the increase of each individual input series within the configured interval, then sums the increases together according to the configured label policy. The output is the cumulative summed increase across all input series.
COUNT: Counts the number of unique input series matched by the configured label policy (for example, cardinality).

The output type of all cumulative counter aggregations is a CUMULATIVE_COUNTER.

`GAUGE`

Gauges support the following aggregation methods:

SUM: Takes the max value of each individual input series within the configured interval, then sums all final values together by the configured label policy.
COUNT: Counts the number of unique input series matched by the configured label policy (for example, cardinality).
MIN: Takes the minimum value of all data points within the configured interval across all series matched by the configured label policy.
MAX: Takes the maximum value of all data points within the configured interval across all series matched by the configured label policy.
PXX, MEAN, MEDIAN, STDEV, SUMSQ: Takes the maximum value of each individual input series within the configured interval, and then computes the value distribution.

The output type of all gauge aggregations is a GAUGE. When querying a gauge metric with a range vector included in the query downsampling might impact the accuracy of the query result. Most use cases that fit this criteria can be converted to use counters instead, which avoids the issue.

`DELTA_COUNTER`

Supported aggregations:

SUM: Sums all values of all series matched by the configured label policy. All values must be nonnegative.
COUNT: Counts the number of unique input series matched by the configured label policy, such as cardinality. Adding more samples to an existing series doesn’t change the count. Only new series impact the count.
COUNT_SAMPLES: Counts the number of input samples matched by the configured label policy. Every sample increments the count, even when multiple samples land on the same series.

The output type of all delta counter aggregations is a DELTA_COUNTER. A DELTA_COUNTER rule treats three settings as a single group: metric_name, aggregation, and the label policy, which is exactly one of keep or discard. Either set all three or omit all three. Setting any one of the three makes the other two required. Omitting all three aggregates matching series as a SUM, keeps the original metric name, and retains every label. The effect is to downsample the counter to the rule’s interval without otherwise changing the series, which is what you want when you need a coarser resolution and nothing else. MEASUREMENT and DELTA_EXPONENTIAL_HISTOGRAM rules can also omit metric_name and the label policy, but both types require an aggregation. Every rollup rule requires a name. The name field identifies the rule itself, and is distinct from metric_name, which names the output metric.

`MEASUREMENT`

A key feature of MEASUREMENT aggregations lies in how they treat individual samples. Unlike other types such as GAUGE and CUMULATIVE_COUNTER, MEASUREMENT metrics aggregate all at once, across all samples of your matching time series within the aggregated time interval. This enables calculation of accurate statistics server-side, within Observability Platform. A typical use case for MEASUREMENT aggregations is calculating statistics across raw request latencies across all instances. This can be correctly performed through metric_type=MEASUREMENT and aggregation=P95. Using metric_type=GAUGE in this scenario produces results you don’t want, discarding all samples except the per-instance max value, then computing the ninety-fifth percentile across these per-instance max values. Every MEASUREMENT rule must set drop_raw to true. Observability Platform rejects a MEASUREMENT rule that retains its raw input data. Measurements support the following aggregation methods:

SUM: Sums all values of all series matched by the configured label policy. All values must be nonnegative. The output metric type is a DELTA_COUNTER.
COUNT_SAMPLES: Counts the number of input samples matched by the configured label policy. The output metric type is a DELTA_COUNTER.
SUMSQ: Sums the squares of all values of all series matched by the configured label policy. The output metric type is a DELTA_COUNTER.
LAST: Takes the last value of all samples matched by the configured label policy. The output metric type is a GAUGE.
MIN: Takes the minimum value of all samples matched by the configured label policy. The output metric type is a GAUGE.
MAX: Takes the maximum value of all samples matched by the configured label policy. The output metric type is a GAUGE.
PXX, MEAN, MEDIAN, STDEV: Computes the value distribution across all samples matched by the configured label policy. The output metric type is a GAUGE.
HISTOGRAM: Summarizes the distribution of values as an exponential histogram with a starting scale of 5. The output type is a DELTA_EXPONENTIAL_HISTOGRAM.

Histograms aggregation operations

If either the input histogram or resulting aggregation exceeds the 160-bucket limit, Observability Platform decreases the exponential histogram scale until the bucket count is within the limit. Downscaling reduces the exponential histogram’s resolution.

`CUMULATIVE_EXPONENTIAL_HISTOGRAM`

Cumulative exponential histogram aggregations operate on OpenTelemetry exponential histograms with cumulative temporality, and on Prometheus native histograms with an exponential bucket layout. Cumulative exponential histograms support this aggregation method:

SUM: Merges input cumulative exponential histograms by the configured label policy. The output metric type is a CUMULATIVE_EXPONENTIAL_HISTOGRAM.

`DELTA_EXPONENTIAL_HISTOGRAM`

Delta exponential histogram aggregations operate on OpenTelemetry exponential histograms with delta temporality. Delta exponential histograms support this aggregation method:

SUM: Merges input delta exponential histograms by the configured label policy. The output metric type is a DELTA_EXPONENTIAL_HISTOGRAM.

Observability Platform

Telemetry Pipeline

More information

View rollup rules

Create a rollup rule

Best practices for rule creation

Chronoctl rollup rule example

Terraform rollup rule example

Delete a rollup rule

Rollup rule attributes

Filters

Aggregation interval

Rule mode

Conflicting output metric names

Label policies

Keep specified labels

Remove specified labels

Set a Graphite label policy

Aggregation operations

`CUMULATIVE_COUNTER`

`GAUGE`

`DELTA_COUNTER`

`MEASUREMENT`

Histograms aggregation operations

`CUMULATIVE_EXPONENTIAL_HISTOGRAM`

`DELTA_EXPONENTIAL_HISTOGRAM`

​View rollup rules

​Create a rollup rule

​Best practices for rule creation

​Chronoctl rollup rule example

​Terraform rollup rule example

​Delete a rollup rule

​Rollup rule attributes

​Filters

​Aggregation interval

​Rule mode

​Conflicting output metric names

​Label policies

​Keep specified labels

​Remove specified labels

​Set a Graphite label policy

​Aggregation operations

​CUMULATIVE_COUNTER

​GAUGE

​DELTA_COUNTER

​MEASUREMENT

​Histograms aggregation operations

​CUMULATIVE_EXPONENTIAL_HISTOGRAM

​DELTA_EXPONENTIAL_HISTOGRAM

View rollup rules

Create a rollup rule

Best practices for rule creation

Chronoctl rollup rule example

Terraform rollup rule example

Delete a rollup rule

Rollup rule attributes

Filters

Aggregation interval

Rule mode

Conflicting output metric names

Label policies

Keep specified labels

Remove specified labels

Set a Graphite label policy

Aggregation operations

`CUMULATIVE_COUNTER`

`GAUGE`

`DELTA_COUNTER`

`MEASUREMENT`

Histograms aggregation operations

`CUMULATIVE_EXPONENTIAL_HISTOGRAM`

`DELTA_EXPONENTIAL_HISTOGRAM`