Derived labels

Derived labels are a construct specifically designed for Chronosphere Observability Platform for efficient operation on individual time series at scale. Use derived labels to simplify the complexity of your labels.

View derived labels

Select from the following methods to view existing derived labels.

Manage derived labels

Create, update, and delete derived labels using Terraform, Chronoctl, or the Observability Platform API.

Create a derived label

The value_glob is the label pattern being matched. This example matches the patterns:

• m3coordinator-read*
• m3coordinator-write*
• m3coordinator-admin*

These values end with an asterisk (*), which matches any pattern. These patterns display under a single derived label defined by the label_name, which is tier.

You can provide multiple definitions for a value with different value_glob patterns. Chronosphere tries them in order of definition.

Delete a derived label

chronoctl derived-labels delete slug_name_1

Use derived labels

In the following examples, the http_requests_total and grpc_requests_total metrics both have a label indicating they’re part of a Kubernetes cluster, but they use different label names. Standardize this label to make it easier for end users to consume. For example, users don’t need to know what label each metric emits when there’s only one standardized label. When a user wants to join these metrics on the cluster label, they can join on the standardized label instead of having to create a complicated query with label_replace.

Mapping derived labels

The following example creates a mapping derived label called cluster, which gets its values from the source label kubernetes_cluster if a metric matches the glob __name__:grpc_*. Similarly, it also gets its values from the source label k8s_cluster if a metric matches the glob __name__:http_*.

This example assumes use these time series as a staring point.

http_requests_total{k8s_cluster="production", method="get", instance="auth-1a2-b3c4"}
http_requests_total{k8s_cluster="canary", method="put", instance="gateway-4s5-9f8b"}
grpc_requests_total{kubernetes_cluster="production", method="get", instance="gateway-0h8-6m2f"}
grpc_requests_total{kubernetes_cluster="canary", method="put", instance="auth-3g8-kl9m"}

api_version: v1/config
kind: DerivedLabel
spec:
  name: cluster mapping label
  slug: cluster-mapping-label
  label_name: cluster
  description: this is a mapping label for cluster
  metric_label:
    mapping_label:
      name_mappings:
        - filters:
            - name: __name__
              value_glob: grpc_*
          source_label: kubernetes_cluster
        - filters:
            - name: __name__
              value_glob: http_*
          source_label: k8s_cluster

If you query http_requests_total{cluster="production"}, the resulting time series is: {__name__="http_requests_total", cluster="production", k8s_cluster="production", method="get", instance="auth-1a2-b3c4"}.

If you query grpc_requests_total{cluster="canary"}, the resulting time series is: {__name__="grpc_requests_total", cluster="canary", kubernetes_cluster="canary", method="put", instance="auth-3g8-kl9m"}

Value mapping for a label that’s both physical and derived

There can be situations where a derived label definition includes a label name that already exists on a metric.

If there’s an existing metric label with the same name as the derived label:

• If existing_label_policy = KEEP, the label that already exists on the metric is used instead of the derived label.
• If existing_label_policy = OVERRIDE, the derived label is used instead of the label that already exists on the metric.
• If existing_label_policy isn’t explicitly set, Chronosphere defaults to the KEEP behavior.

The following examples start with this set of time series:

{__name__="grpc_metric", service="labrador", instance="blah"}
{__name__="grpc_metric", service="golden", instance="blah"}
{__name__="envoy_metric", microservice="Labrador", instance="blah"}
{__name__="envoy_metric", microservice="GOLDEN", instance="blah"}

Standardize label names

Different teams or services provide source data, some of which might not conform to the intended labeling standard. For cleaner reviews, you can map all of the provided into a standardized space. Nothing can be added to the target space without explicit mapping, which preserves and augments the original labeling.

For example:

Top-level value mappings

Top-level value mappings are mappings that apply to all name mappings, and not to a specific name mapping. This MappingLabel includes top-level value mappings in the value_mappings section.

label_name: standard_service
mapping_label:
  name_mappings:
     - filters:
        - name: __name__
          value_glob: grpc_*
       source_label: service
     - filters:
        - name: __name__
          value_glob: envoy_*
       source_label: microservice
  value_mappings:
    - source_value_globs:
       - labrador
       - Labrador
      target_value: labrador-retriever
    - source_value_globs:
       - golden
       - GOLDEN
      target_value: golden-retriever

If you were to use this query:

grpc_metric{standard_service="labrador-retriever"}

The resulting time series is:

{__name__="grpc_metric", service="labrador", standard_service="labrador-retriever", instance="blah"}

For this query:

envoy_metric{standard_service="labrador-retriever"}

The resulting time series is:

{__name__="envoy_metric", microservice="Labrador", standard_service="labrador-retriever", instance="blah"}

If you query using a non-existent target value such as:

grpc_metric{standard_service="labrador"}

The query returns this time series:

{__name__="grpc_metric", service="labrador", standard_service="labrador-retriever", instance="blah"}

Value mappings inside a name mapping

Value mappings match to one specific name mapping.

For example:

label_name: standard_service
  mapping_label:
    name_mappings:
       - filters:
          - name: __name__
            value_glob: grpc_*
         source_label: service
         value_mappings:
          - source_value_globs:
             - labrador
             - Labrador
            target_value: labrador-retriever
       - filters:
          - name: __name__
            value_glob: envoy_*
          source_label: microservice

If you query grpc_metric{standard_service="labrador-retriever"}, with this configuration, the target value labrador-retriever applies only to metrics that match __name__:grpc_*.

The resulting time series is:

{__name__="grpc_metric", service="labrador", standard_service="labrador-retriever", instance="blah"}

Consider the query envoy_metric{standard_service="labrador-retriever"}. According to the configuration, the value mapping for labrador-retriever applies only to metrics that match __name__:grpc_*.

This query returns no results because:

• standard_service="labrador-retriever" isn’t a physical label-value pair on envoy_metric.
• There is no top-level value mapping where labrador-retriever is the target value. There’s also no name mapping specific value mapping where labrador-retriever is the target value for metrics that match __name__:envoy_*.

Value mapping inside name mapping with top level mapping together

You can use Value Mappings and name mapping level value mappings in the same configuration. For example:

label_name: standard_service
  mapping_label:
    name_mappings:
       - filters:
          - name: __name__
            value_glob: grpc_*
         source_label: service
         value_mappings:
          - source_value_glob:
             - labrador
             - Labrador
            target_value: labrador-retriever
       - filters:
          - name: __name__
            value_glob: envoy_*
          source_label: microservice
    value_mappings:
    - source_value_globs:
       - labrador
       - Labrador
      target_value: golden-retriever

In this example, target value labrador-retriever applies specifically to metrics that matches __name__:grpc_* because it matches to that specific name mapping, where value labrador-retriever applies to all name mappings.

For the query grpc_metric{standard_service="labrador-retriever"} results in the time series:

{__name__="grpc_service", service="labrador", standard_service="labrador-retriever", instance="blah"}

Similarly, for the query envoy_metric{standard_service="golden-retriever"}, the resulting time series is:

{__name__="envoy_metric", microservice="GOLDEN", standard_service="golden-retriever", instance="blah"}

However, querying envoy_metric{standard_service="labrador-retriever"} returns no values, because:

• standard_service="labrador-retriever" isn’t a physical label-value pair on envoy_metric.
• There is no top-level value mapping where labrador-retriever is the target value. There’s also no name mapping that has a value mapping where labrador-retriever is the target value for metrics that match __name__:envoy_*.

If there is a top-level value mapping with the same target value as a name mapping specific value mapping, like this situation:

label_name: standard_service
  mapping_label:
    name_mappings:
       - filters:
          - name: __name__
            value_glob: grpc_*
         source_label: service
         value_mappings:
          - source_value_filters:
             - labrador
             - Labrador
            target_value: golden-retriever
       - filters:
          - name: __name__
            value_glob: envoy_*
          source_label: microservice
    value_mappings:
     - source_value_globs:
        - golden
        - GOLDEN
       target_value: golden-retriever

In this case, both value mappings have the same target value of golden-retriever. The value mappings that are name mapping specific apply over the top-level value mappings.

Constructed derived labels

As an example, you might want to query each of these metrics by the service which the HTTP requests and gRPC requests originated from. However there’s no service label, but there is an instance label that has the name of the server instance the requests came originated from. Chronosphere experience is that this happens rarely, and recommends that you use mapped derived labels whenever possible.

This example creates a constructed derived label called service for a metric if the metric matches the glob instance:auth-*, with service label with the value auth. Similarly, it creates the label service for a metric if this metric matches the glob instance:gateway-*, and this service label has the value gateway.

This example assumes the use of these time series as a staring point:

http_requests_total{k8s_cluster="production", method="get", instance="auth-1a2-b3c4"}
http_requests_total{k8s_cluster="canary", method="put", instance="gateway-4s5-9f8b"}
grpc_requests_total{kubernetes_cluster="production", method="get", instance="gateway-0h8-6m2f"}
grpc_requests_total{kubernetes_cluster="canary", method="put", instance="auth-3g8-kl9m"}

api_version: v1/config
kind: DerivedLabel
spec:
  name: Service constructed label
  slug: service-constructed-label
  label_name: service
  description: this is a constructed derived label for service
  metric_label:
    constructed_label:
      value_definitions:
      - value: auth
        filters:
          - name: instance
            value_glob: auth-*
      - value: gateway
        filters:
          - name: instance
            value_glob: gateway-*

For example, if you query the http_requests_total{service="auth"} label, the resulting time series is:

{__name__="http_requests_total", k8s_cluster="production", method="get", instance="auth-1a2-b3c4", service="auth"}

If you query the grpc_requests_total{service="gateway"} label, the resulting time series is:

{__name__="grpc_requests_total", kubernetes_cluster="production", method="get", instance="gateway-0h8-6m2f", service="gateway"}