Ingest DogStatsD metrics with the OpenTelemetry Collector

The OpenTelemetry Collector StatsD receiver (opens in a new tab) accepts, transforms, and aggregates DogStatsD metrics into OpenTelemetry protocol (OTLP) metrics for use in the Chronosphere Observability Platform.

Install the OpenTelemetry Collector

To install the OpenTelemetry Collector, deploy it as a Kubernetes DaemonSet on the same node as your DogStatsD clients.

DogStatsD clients use the User Datagram Protocol (UDP) to send metrics. The OpenTelemetry Collector must be on the same node as the DogStatsD client to ensure reliable network communication between the DogStatsD client and the OpenTelemetry Collector.

Configure the OpenTelemetry StatsD receiver

You can configure the StatsD receiver by adding it to your OpenTelemetry Configuration (opens in a new tab).

The following configuration defines an OpenTelemetry receiver and a StatsD receiver (opens in a new tab) to accept DogStatsD metrics from Datadog clients.

````suggestion
receivers:
  otlp:
    protocols:
      grpc:
 
  statsd/dogstatsd:
    # Default UDP port.
    endpoint: "localhost:8125"
    # Default interval. Use your licensed resolution: 60s, 30s or 15s.
    aggregation_interval: 60s
    # Metric type must be enabled. Default.
    enable_metric_type: true
    # DogStatsD clients support key-only tags, but attributes without a value are ignored in the Prometheus data model.
    enable_simple_tags: false
    # DogStatsD clients send delta counters (non-monotonic counters).
    is_monotonic_counter: true
    enable_ip_only_aggregation: true
    # Aggregates DogStatsD timers as exponential histograms.
    timer_histogram_mapping:
      - statsd_type: "timing"
        observer_type: "histogram"
        histogram:
          max_size: 160
      - statsd_type: "histogram"
        observer_type: "histogram"
        histogram:
          max_size: 160
      - statsd_type: "distribution"
        observer_type: "histogram"
        histogram:
          max_size: 160

Configure an OpenTelemetry protocol exporter to send DogStatsD metrics

DogStatsD metrics sent to Datadog are normalized to follow Datadog's best practices (opens in a new tab). For example, Datadog lowercases tag names and values and modifies tag values to begin with an alphabetical character.

To reduce migration efforts and minimize changes to queries, Observability Platform can apply the same sanitization to ensure metric names, tags, and values are the same in both Observability Platform and Datadog by passing an additional header.

The following configuration defines two OpenTelemetry protocol (OTLP) exporters:

otlp/chronosphere defines a default configuration to send OTLP metrics without applying Datadog normalization. Use this exporter to send all metrics from any source except Datadog clients.
otlp/chronosphere/dogstatsd defines a configuration to send OTLP metrics and apply Datadog normalization. Use this exporter only to send metrics that were sent by Datadog clients.

exporters:
  # Exporter configured without normalization
  otlp/chronosphere:
    endpoint: ADDRESS.chronosphere.io:443
    timeout: 30s
    retry_on_failure:
      enabled: true
    compression: zstd
    headers:
      API-Token: $API_TOKEN
  # Exporter configured to apply Datadog metric normalization
  otlp/chronosphere/dogstatsd:
    endpoint: ADDRESS.chronosphere.io:443
    timeout: 30s
    retry_on_failure:
      enabled: true
    compression: zstd
    headers:
      API-Token: $API_TOKEN
      # Enable Datadog metric name and tag normalization
      Chronosphere-Metrics-Translation: datadog
      Chronosphere-Metrics-Translation-Datadog-NormalizeLabelValues: "true"

To configure these exporters for your instance:

Replace ADDRESS with the company name subdomain in your Observability Platform instance address ending in .chronosphere.io:443. For example, if your instance's domain is MY_COMPANY.chronosphere.io:443, replace ADDRESS with MY_COMPANY.
Set $API_TOKEN to the API token generated from your Chronosphere service account, specified as an HTTP header. Chronosphere recommends calling this value as an environment variable.
Set retry_on_failure to true to retry attempts after any recoverable errors.
Set timeout to 30s to prevent larger requests from timing out, since the upstream system might require more time for internal batching.

Define OpenTelemetry Collector pipelines

To receive metrics, connect the receivers and exporters with two separate pipelines:

metrics pipeline: The default metrics pipeline receives all OTLP metrics and sends them to the Chronosphere OTLP endpoint.
metrics/dogstatsd pipeline: The DogStatsD metrics pipeline receives DogStatsD metrics and exports them to a Chronosphere endpoint with a header to enable Datadog metric name and tag normalization.

The following service configuration enables the StatsD receiver for the appropriate pipeline:

service:
  pipelines:
    # Defines the default metrics pipeline.
    metrics:
      receivers: [otlp]
      processors: [batch, resourcedetection, resource/service-instance]
      exporters: [otlp/chronosphere]
    # Defines a metrics pipeline with Datadog metric normalization.
    metrics/dogstatsd:
      receivers: [otlp/chronosphere/dogstatsd]
      processors: [batch, resourcedetection, resource/service-instance]
      exporters: [otlp/chronosphere/dogstatsd]

The processors value adds processors to define batching and ensure time series uniqueness.

batch: Sets batch processing. For more information, see the "Configure batch processing" step in Configure the OpenTelemetry Collector.
resourcedetection: Collects information about the environment in which the OpenTelemetry Collector is running.
resource/service-instance: Observability Platform requires a service.instance.id attribute and value for all time series to ensure time series uniqueness. The ingestion API rejects metrics that lack service.instance.id. For more information, see the Map resource attributes.

OTLP endpoints Chronosphere Collector