Derived metrics - Chronosphere Documentation

Derived metrics let you create user-friendly names for queries. Use derived metrics to implement query aliasing, which reduces the need to write complex queries.

View your derived metrics

Chronoctl
API

View derived metrics with the Chronoctl command chronoctl derived-metrics list, filtered by their slugs with the --slugs flag.For example, to list all derived metrics:

chronoctl derived-metrics list

To list derived metrics with slugs slug_name_1 and slug_name_2:

chronoctl derived-metrics list --slugs slug_name_1,slug_name_2

To complete this action with the Chronosphere API, use the ListDerivedMetric endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Create a derived metric

Chronoctl
Terraform
API

Here’s a Chronoctl example of a derived metric with two underlying expressions:

api_version: v1/config
kind: DerivedMetric
spec:
  name: my derived metric
  slug: my-derived-metric
  metric_name: test_metric
  description: This is a test derived metric
  queries:
  - query:
      prometheus_expr: scrape_duration_seconds{$job}
      variables:
        - name: job
          default_prometheus_selector: job=~".*"
    selector:
      labels:
      - name: id
        type: EXACT
        value: abc
  - query:
      prometheus_expr: scrape_series_added{$job}
      variables:
        - name: job
          default_prometheus_selector: job=~".*"

Create a derived metric with Terraform by using the chronosphere_derived_metric type followed by a name in the resource declaration:

resource "chronosphere_derived_metric" "my-derived-metric" {
    # The name of the derived metric.
    name        = "my derived metric"
    # Slug is the unique identifier of the derived metric.
    slug        = "my-derived-metric"
    # Description about the derived metric.
    description = "This is a test derived metric"
    # Metric name to be used to query for the derived metric.
    metric_name = "test_metric"

    # List of queries corresponding to the derived metric. If more than one query is specified,
    # all but one need to have selectors defined.
    queries {
        # Optional selector indicates which query to select for a derived metric based on the selector.
        # For example, the selector will match a query like test_metric{id="abc"}.
        selector {
            # The set of labels that need to match for the selector to be considered a match.
            # The value here can be a regex value.
            labels = {
                id = "abc"
            }
        }
        # The query corresponding to the previously described selector.
        query {
            # expr has the expression meant to be run for the derived metric, and must contain the
            # list of selector variables that will be applied / propagated from the derived metrics query.
            # Selectors not specified in the query apply after the query has been processed in a post-processing step.
            expr = "scrape_duration_seconds{$job}"

            # variables contains information about selectors used in the expression. Each selector specified in the
            # expression must have a variables block.
            variables {
                # The name of the variable.
                name             = "job"
                # The default selector expression to fill in the query if selector isn't provided
                # on the derived metrics query.
                default_selector = "job=~\".*\""
            }
        }
    }

    # An example of another query without any selectors. Note how it points to a different expression.
    queries {
        query {
            expr = "scrape_series_added{$job}"
            variables {
                name             = "job"
                default_selector = "job=~\".*\""
            }
        }
    }
}

To complete this action with the Chronosphere API, use the CreateDerivedMetric endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Use a selector in your derived metric definition when you want a metric name that’s used by different queries based on the selector, or when you want the same derived metric to map to different underlying metrics. This can cause performance issues if you have a large number of id items in use. Chronosphere doesn’t support sums across id items. If you have many selectors, a recording rule is often a better option. If you want to map only a derived metric name to a query, you don’t need a selector.

Pass through query matchers

The special variable $__passthrough_matchers forwards all label matchers from the caller’s query into the underlying metric expression. Use it when you want the derived metric to accept arbitrary label filters without enumerating every potential label as a named variable. Add $__passthrough_matchers as a selector inside the curly braces of the underlying metric in the expression field. For example:

Chronoctl
Terraform

api_version: v1/config
kind: DerivedMetric
spec:
  name: request errors
  slug: request-errors
  metric_name: http_request_errors_total
  queries:
    - query:
      prometheus_expr: http_requests_total{$__passthrough_matchers, status=~"5.."}

resource "chronosphere_derived_metric" "request-errors" {
  name        = "request errors"
  slug        = "request-errors"
  metric_name = "http_request_errors_total"

  queries {
    query {
      expr = "http_requests_total{$__passthrough_matchers, status=~\"5..\"}"
    }
  }
}

When you query http_request_errors_total{env="prod", region="us-east-1"}, Observability Platform expands the expression to http_requests_total{env="prod", region="us-east-1", status=~"5.."} before executing it. If you provide no matchers, query results are unaffected because the variable expands to a query that matches all series. You can combine $__passthrough_matchers with named variables and fixed label matchers in the same expression:

http_requests_total{$__passthrough_matchers, $service, status=~"5.."}

$__passthrough_matchers: A variable that expands to every matcher that the caller supplies.
$service: A named variable with its own default selector. For an example, see Create a derived metric.
status=~"5..": A fixed matcher that’s always added to the query.

The difference between $__passthrough_matchers and a named variable such as $job is its scope. Named variables match one specific label, while $__passthrough_matchers captures everything the caller provides, regardless of which labels they filter.

Delete a derived metric

Users can modify Terraform-managed resources only by using Terraform. Learn more.

Chronoctl
Terraform
API

To delete a derived metric with Chronoctl, use the chronoctl derived-metrics delete command with the slug of the derived metric you want to delete:

chronoctl derived-metrics delete SLUG_NAME

You can delete more than one metric at a time by providing a comma-separated list of slugs. For example:

chronoctl derived-metrics delete SLUG_NAME_1,SLUG_NAME_2

To delete a derived metric with Terraform:

Remove the derived metric’s definition from the Terraform file.
Run terraform apply.

To complete this action with the Chronosphere API, use the DeleteDerivedMetric endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Replace a recording rule

When you have a complex or slow recording rule, in some cases you can replace the rule with a derived metric. For example, this recording rule queries for a number of HTTP status codes:

- record: slo:sli_error:ratio_rate5m
  expr: |  (sum(rate(flask_http_request_duration_seconds_count{job="default/productservice-servicemonitor/0", status=~"(5..|4..)"}[5m])))
    /    (sum(rate(flask_http_request_duration_seconds_count{job="default/productservice-servicemonitor/0"}[5m])))
  labels:
    owner: customersuccess
    repo: chronosphereio/productservice
    sloth_id: productservice-requests-availability
    sloth_service: productservice
    sloth_slo: requests-availability
    sloth_window: 5m
    tier: "2"

Replacing the recording rule with this derived metric can reduce query load:

resource "chronosphere_derived_metric" "slo-error-rate-5m" {
  name     = "slo-error-rate-5m"
  slug     = "slo-error-rate-5m"
  description = "Service Error Rate - 5m"
  metric_name = "slo:sli_error:ratio_rate5m"

  queries {
     query {
        expr = """
         sum(rate(flask_http_request_duration_seconds_count{
             $job,
             status=~"(5..|4..)"
         }[1m]))
         /
         sum(rate(flask_http_request_duration_seconds_count{
             $job
         }[1m]))
       """
   variables {
        name = "job"
        default_selector = "job=~\".*\""
   }
    }
  }
}

​View your derived metrics

​Create a derived metric

​Pass through query matchers

​Delete a derived metric

​Replace a recording rule

View your derived metrics

Create a derived metric

Pass through query matchers

Delete a derived metric

Replace a recording rule