Derived labels

Metrics from different sources can use the same labels to mean different things. For example, MySQL metrics could use the cluster label for a service it provides, gRPC based application metrics use grpc_service, while the rest of your applications use standard-service as a label for one or more processes. From the end user's perspective, these all mean service. Sometimes you need a combination of variables to determine if these are the same information from different applications. Maybe the staging MySQL cluster hosts many services, but production has a dedicated cluster per service.

Relabel rules are the language that Prometheus (opens in a new tab) provides to tune scraping, determine which time series to persist, and modify a time series before persisting it. To modify a time series, you can use relabel rules to update a metric's target_label or to update multiple labels. Relabel rules overwrite existing labels, removing labels previously associated with a metric.

Chronosphere exposes this construct directly in the Collector. Relabel rules are, by design, metric-centric. To make a change to a particular label on all metrics, edit the relabel rules for every scrape job.

Derived labels are a Chronosphere-specific construct. Unlike relabel rules, they are non-destructive and are specifically designed for efficient operation on time series at scale.

Instead of using regular expressions for matching the time series to operate on, derived labels use the same glob matching expressions used by drop rules, aggregation rules, and traffic shaping pools.

Use derived labels to augment a time series after it has been persisted. Derived labels are non-destructive, meaning they don't make permanent changes to the underlying time series. If you remove a derived label, the label goes away, and the underlying time series remain. A relabel rule permanently changes the labels applied to a time series however, and can't be undone.

Differences between relabeling and deriving

When to use relabel versus derived

If you're not sure whether to use relabeling, or to use a derived label, use the following guidelines to help you decide:

Relabel Rules

• You want to remove existing labels and replace them with one or more new labels.
• You need to drop time series and scrape targets.
• You want to control configuration at the Collector. For example, you want to edit the configuration for a single service using a service monitor.
• You need to control data sent to Chronosphere. For example, dropping data to save network cost.
• You want to do a complex label modification operation, like using arbitrary regular expressions with capture groups.

Derived Labels

• You want to retroactively change the labels on a previously emitted time series in a non-destructive way.
• When fixing the source or scrape location is difficult. For instance, if the data source is in a customer environment, or changing scrape configuration is prohibitively expensive in your environment.
• When you want to manage the label configuration in a label-centric way. For example, if you want to add a label to all of your metrics with some value based on the source labels, you have to change the scrape configuration for every service.

Use derived labels

In the following examples, the http_requests_total and grpc_requests_total metrics both have a label indicating they're part of a Kubernetes cluster, but they use different label names. Standardize this label to make it easier for end users to consume. For example, users don't need to know what label each metric emits when there's only one standardized label. When a user wants to join these metrics on the cluster label, they can join on the standardized label instead of having to create a complicated query with label_replace.

The examples assume use these time series as a staring point.

http_requests_total{k8s_cluster="production", method="get", instance="auth-1a2-b3c4"}
http_requests_total{k8s_cluster="canary", method="put", instance="gateway-4s5-9f8b"}
grpc_requests_total{kubernetes_cluster="production", method="get", instance="gateway-0h8-6m2f"}
grpc_requests_total{kubernetes_cluster="canary", method="put", instance="auth-3g8-kl9m"}

Mapping derived labels

The following example creates a mapping derived label called cluster, which gets its values from the source label kubernetes_cluster if a metric matches the glob __name__:grpc_*. Similarly, it also gets its values from the source label k8s_cluster if a metric matches the glob __name__:http_*.

api_version: v1/config
kind: DerivedLabel
spec:
  name: cluster mapping label
  slug: cluster-mapping-label
  label_name: cluster
  description: this is a mapping label for cluster
  metric_label:
    mapping_label:
      name_mappings:
        - filters:
            - name: __name__
              value_glob: grpc_*
          source_label: kubernetes_cluster
        - filters:
            - name: __name__
              value_glob: http_*
          source_label: k8s_cluster

Usage

If you query http_requests_total{cluster="production"}, the resulting time series is: {__name__="http_requests_total", cluster="production", k8s_cluster="production", method="get", instance="auth-1a2-b3c4"}.

If you query grpc_requests_total{cluster="canary"}, the resulting time series is: {__name__="grpc_requests_total", cluster="canary", kubernetes_cluster="canary", method="put", instance="auth-3g8-kl9m"}

Constructed derived labels

As an example, you might want to query each of these metrics by the service which the HTTP requests and gRPC requests originated from. However there's no service label, but there is an instance label that has the name of the server instance the requests came originated from.

This example creates a constructed derived label called service for a metric if the metric matches the glob instance:auth-*, with service label with the value auth. Similarly, it creates the label service for a metric if this metric matches the glob instance:gateway-*, and this service label has the value gateway.

api_version: v1/config
kind: DerivedLabel
spec:
  name: Service constructed label
  slug: service-constructed-label
  label_name: service
  description: this is a constructed derived label for service
  metric_label:
    constructed_label:
      value_definitions:
      - value: auth
          filters:
            - name: instance
              value_glob: auth-*
        - value: gateway
          filters:
            - name: instance
              value_glob: gateway-*

Usage

If you query http_requests_total{service="auth"}, the resulting time series is: {__name__="http_requests_total", k8s_cluster="production", method="get", instance="auth-1a2-b3c4", service="auth"}.

If you query grpc_requests_total{service="gateway"}, the resulting time series is: {__name__="grpc_requests_total", kubernetes_cluster="production", method="get", instance="gateway-0h8-6m2f", service="gateway"}.

Value mapping for a label that's both physical and derived

There can be situations where a derived label definition includes a label name that already exists on a metric.

If there's an existing metric label with the same name as the derived label:

• If existing_label_policy = KEEP, the label that already exists on the metric is used instead of the derived label.
• If existing_label_policy = OVERRIDE, the derived label is used instead of the label that already exists on the metric.
• If existing_label_policy isn't explicitly set, Chronosphere defaults to the KEEP behavior.