Head sampling

Head sampling

You can configure head sampling (opens in a new tab) to capture only a portion of possible traces from originating services. Use head sampling rules to limit possible traces to a fraction of all possible traces. You define sampling strategies to target specific kinds of traces, focus on specific services, or concentrate on a combination of service and operation.

View head sampling rules

You can view head sampling rules in the Chronosphere app to understand the impact of each rule on your tracing data, such as the sampling rate, rule criteria, and the impact the rule is having on your trace data volume.

To return a list of defined head sampling rules without additional information, use Chronoctl.

In the Chronosphere app, each head sampling rule indicates which service or combination of service and operation it impacts. A single rule can impact multiple operations at different sampling rates for a specific service.

To view head sampling rules:

  1. In the navigation menu select Shaping > Trace Control Plane.

  2. Click the Head sampling tab. The Status column indicates whether each rule is active or inactive. The Traces Kept column indicates the percentage at which the current rule is sampling your traces for the indicated service.

    Hold the pointer over the bar in the Traces Kept column to view a description for each rule.

  3. Expand each of the configured rules to view the rule criteria and impact on your tracing data. You can also view which operations the rule impacts for the selected service.

  4. If a rule impacts multiple operations, use the dropdown menu to select which operation to display impact data for.

  5. Use the search box to locate rules impacting a specific service.

Configure head sampling

To instrument head sampling, Chronosphere supports the OpenTelemetry JaegerRemoteSampler (opens in a new tab) head sampling standard. Chronosphere implements the JaegerRemoteSampler configuration API (opens in a new tab), and serves sampling strategies based on the Terraform resources you define.

Chronosphere is compatible with the Jaeger Remote Sampling extension (opens in a new tab) to enable the OpenTelemetry Collector to act as a proxy between OpenTelemetry Agents and the Chronosphere backend.

To configure head sampling:

  1. Configure the OpenTelemetry Collector.
  2. Configure OpenTelemetry Agents.
  3. Manage your strategy with Terraform.

Configure the OpenTelemetry Collector

Before configuring OpenTelemetry Agents, you must configure the OpenTelemetry Collector to export tracing data to Chronosphere.

Chronosphere requires version 0.83 or later of the OpenTelemetry Collector.

  1. Configure the OpenTelemetry Collector to export tracing data to Chronosphere.

  2. In the OpenTelemetry Collector config.yaml file, apply the following settings to modify the extensions YAML collection. Specify an endpoint that points to your Chronosphere tenant, and include the Chronosphere API token you created for the OpenTelemetry Collector.

    Chronosphere recommends specifying a reload_interval to control a local cache of your sampling strategy for the remote source (which is Chronosphere). This interval reduces the number of backend API calls the OpenTelemetry Collector makes for your instrumented Agents, which helps reduce egress costs.

      health_check: {}
      pprof: {}
          reload_interval: 30s
            endpoint: ADDRESS
            compression: gzip
              API-Token: API_TOKEN
    • ADDRESS: Your company name prefixed to your Chronosphere instance that ends in .chronosphere.io:443. For example, MY_COMPANY.chronosphere.io:443.
    • API_TOKEN: The API token generated from your service account. Chronosphere recommends storing your API token in a separate file or Kubernetes Secret and calling it using an environment variable, such as $API_TOKEN.

Configure OpenTelemetry Agents

You configure OpenTelemetry Agents to act as a proxy for the OpenTelemetry Collector, which sends your tracing data to Chronosphere. When configuring this sampler, you must include all of these properties:

  • Endpoint: Endpoint where your OpenTelemetry Collector is running, which includes the Jaeger Remote Sampling extension that points to Chronosphere.
  • Polling interval: Interval at which your OpenTelemetry Agents sync strategies from your OpenTelemetry Collector.
  • Initial sampler: Initial policy to implement from your service until the configured endpoint pulls a strategy.

The implementation depends on the programming language of your app. For example, the following code from the Go implementation of OpenTelemetry, OpenTelemetry-Go (opens in a new tab), defines how to implement the JaegerRemoteSampler in Go:

jaegerRemoteSampler := jaegerremote.New(
tp := trace.NewTracerProvider(
  • SERVICE_NAME: Name of the service you're sampling, which can map to a microservice in your architecture.
  • HOST_NAME: Host name where your OpenTelemetry Collector is running.

Create head sampling rules

After configuring the OpenTelemetry Collector to export traces, configuring head sampling, and configuring your OpenTelemetry Agents, you can manage your head sampling strategy using the Chronosphere Terraform provider or Chronoctl. This ability means you can push strategy changes to all of your OpenTelemetry Agents without modifying the JaegerRemoteSampler strategy directly.

If you don't define a sampling strategy for a service, Chronosphere applies the default sampling rate of 0.001 (0.1%) to the service.

The applied_strategy defines your sampling strategy, and can be one of the following values:

  • probabilistic_strategy: Defines a probabilistic strategy, which samples traces from the identified service based on the sampling_rate. This value determines the probability of sampling any trace, and must be in the range of 0 to 1.
  • rate_limiting_strategy: Defines a rate-limited strategy that sets the maximum number of traces to sample per second.
  • per_operation_strategies: Defines a probabilistic strategy that sets a default sampling rate, plus an upper and lower bound.

To manage your head sampling strategy with Chronoctl, install Chronoctl version 0.59.0 or later.

When creating your head sampling strategy with Chronoctl, consider the following behaviors of the slug property:

  • If you don't provide a slug, Chronoctl generates one based on the name field.
  • After creating your head sampling strategy, you can modify the name but can't modify the slug.
  • The service_name and slug must match.

To define your head sampling strategy:

  1. Create a YAML file and define your head sampling strategy.

    Use the chronoctl trace-jaeger-remote-sampling-strategies scaffold command to generate an example resource for a head sampling rule.

    The following head sampling strategy defines a probabilistic strategy that sets a default sampling rate for the inventory-operation, plus an upper and lower bound:

    api_version: v1/config
    kind: TraceJaegerRemoteSamplingStrategy
        slug: ordering-service-sampling
        name: Ordering service head sampling strategy
        service_name: ordering-service-sampling
                default_sampling_rate: 0.01
                default_lower_bound_traces_per_second: 1
                default_upper_bound_traces_per_second: 1000
                      operation: inventory-operation
                        sampling_rate: 0.1
  2. Apply your head sampling strategy and send it to Chronosphere:

    chronoctl apply -f {FILE_NAME}.yml

    FILE_NAME is the name of your head sampling YAML file.

Terraform example

The following example provides a resource definition for three services and defines a distinct sampling strategy for each:

  • The first resource samples the billing-service service at a rate of 0.01, which is 1% of traces.
  • The second resource samples at most two traces per second, per instance, from the cart-service service. For example, if the cart-service service consists of 17 pods, the expected sample of traces per second is somewhere between 0 and 34.
  • The third resource samples traces from the ordering-service service at a rate of 0.01, or 1% of traces. If volumes are low, Chronosphere samples traces at least once per second. If volumes are high, Chronosphere stops sampling traces after reaching 1,000 traces per second.
resource "chronosphere_trace_jaeger_remote_sampling_strategy" "billing-service" {
  name = "billing-service Jaeger Remote Sampling strategy"
  service_name = "billing-service"
  applied_strategy {
    probabilistic_strategy {
      sampling_rate = 0.01
resource "chronosphere_trace_jaeger_remote_sampling_strategy" "cart-service" {
  name = "cart-service Jaeger Remote Sampling strategy"
  service_name = "cart-service"
  applied_strategy {
    rate_limiting_strategy {
      max_traces_per_second = 2
resource "chronosphere_trace_jaeger_remote_sampling_strategy" "ordering-service" {
  name = "ordering-service Jaeger Remote Sampling strategy"
  service_name = "ordering-service"
  applied_strategy {
    per_operation_strategies {
      default_sampling_rate = 0.01
      default_lower_bound_traces_per_second = 1
      default_upper_bound_traces_per_second = 1000
      per_operation_strategies {
        operation = "notification-operation"
        probabilistic_strategy {
          sampling_rate = 0.0
      per_operation_strategies {
        operation = "inventory-operation"
        probabilistic_strategy {
          sampling_rate = 0.1
      per_operation_strategies {
        operation = "payment-operation"
        probabilistic_strategy {
          sampling_rate = 1.0