DogStatsD ingestion

When accepting DogStatsD metrics over the UDP, Observability Platform requires you to run the Chronosphere Collector as a Kubernetes DaemonSet or as a sidecar.

All ingested DogStatsD metrics are subject to sanitization rules that follow the Datadog best practice guidelines (opens in a new tab). For example, Chronosphere Observability Platform converts all DogStatsD labels to lowercase.

For information on converting Datadog metrics and dashboards to Observability Platform, see the Datadog overview.

Deploy the Chronosphere Collector

Datadog metrics libraries primarily send metrics using the UDP protocol, which sends packets without reliability mechanisms, such as handshakes, retries, or acknowledgments. Using UDP to send data over the network increases the risk of losing metric data due to lost packets and other network issues.

Because local UDP communication is generally more stable, keeping communication on the same Kubernetes node reduces this risk.

Install Collector as a Kubernetes DaemonSet

To install the Chronosphere Collector, follow the steps described in Kubernetes Collector installation to retrieve metrics.

The manifest defines a ClusterRole with the permissions required to access the local Kubelet API and mounts a volume to access the cgroups directory on the node.

Configure DogStatsD ingestion

1. Use the configuration from the annotated manifest (opens in a new tab) in the Kubernetes Collector installation documentation as a starting point.

2. Add hostPort to the DaemonSet containters.ports YAML collection to enable network communication on the host network.

   spec:
     template:
       spec:
         containers:
           ports:
           ...
           - containerPort: 8125
             hostPort: 8125  # Add hostPort
             name: dogstatsd
             protocol: UDP

3. Enable DogStatsD mode under the push.dogstatsd.enabled YAML collection in the Collector manifest.

   push:
      dogstatsd:
         enabled: true

4. Modify additional DogStatsD configuration options under the push.dogstatsd YAML collection.

DogStatsD configuration options

You can configure the following options in the Collector manifest for DogStatsD.

• listenAddress: The address the server listens on. The default is 0.0.0.0:8125. Your DogStatsD client points to this address and port.

  When changing the default port, also update the container ports in the DaemonSet manifest to match.

• listenProtocol: The protocol the server listens on. The Chronosphere Collector supports udp, tcp, or tcp/udp (both protocols on the same port). The default is udp.

  push:
     dogstatsd:
        enabled: true
        listenAddress: 0.0.0.0:8125
        listenProtocol: udp

• aggregation: A YAML collection that defines how the Collector ingests a data sample. The Collector aggregates the data based on the metric type and sends a single data point to Observability Platform that represents the sample for the defined interval. These aggregations reduce network egress and processed writes to Observability Platform.

  The Collector aggregates data points based on the DogStatsD metric type as follows:

  • counters: Sums all data point values for a time series and sends the SUM as a DELTA COUNTER. Not configurable.
  • gauges: Selects the LAST value for a time series in the interval. Not configurable.
  • timers: Aggregates timer values into a DELTA EXPONENTIAL HISTOGRAM, by default, including DogStatsD histograms and distributions.

    • aggregationType: Defines how the Collector aggregates timers. Defaults to histogram.

      The deprecated tdigest option provides backward compatibility for users of DogStatsD support in Collector versions prior to v0.110.0, and will be removed in future versions. Unlike histograms, t-digests require additional aggregation rules to calculate and persist a computed quantile, count, min, max, or average as a gauge in Observability Platform.

  • interval: Defines the length of time before writing to Observability Platform. Set an interval value that aligns with your licensed persistent writes and cardinality ratio. A more frequent aggregation interval increases persisted writes.
  • inactiveExpireAfter: Determines the length of time before Chronosphere evicts unused aggregation keys from the local cache. Defaults to 2m.

• labels: Adds labels to all DogStatsD metrics pushed. Define each label as a key-value pair.

  For example, the default configuration adds the labels env and k8s_cluster_name using environment variables for the values.

• prefix: Adds a prefix to all DogStatsD metrics pushed.

  For example, setting this value to qa prefixes all metric names with qa followed by `.`, the dot separator. A metric like build_info.ip becomes qa.build_info.ip and is normalized to qa_build_info_ip.

• kubernetesMetadataAugmentation: Defines the configuration and addition of Kubernetes metadata, labels, and annotations to be added to the time series.

  The Chronosphere Collector doesn’t support using node or namespace metadata as the source for label augmentation.

  • enabled: A Boolean value that toggles augmentation. Defaults to false. The Chronosphere Collector retrieves metadata from the local Kubelet API and doesn’t support other sources.

  • kubelet:

    • kubeletNodeIP: Defines the public IP address assigned to Kubelet. Defaults to the value of the KUBERNETES_NODE_IP environment variable.
    • kubeletNodePort: Defines the port that Kubelet listens on for HTTP requests. Defaults to 10250.
    • kubeletNodePodsEndpoint: Overrides the URL of Kubelet’s /pods endpoint, which by default is constructed from kubeletNodePort and kubeletNodeIP as https://${kubeletNodePort}:${kubeletNodeIP}/pods.
    • kubeletNodeTLSInsecureSkipVerify: Determines whether the Chronosphere Collector skips verification of Kubelet’s TLS certificate. Defaults to true.
    • kubeletNodeBearerTokenFile: Defines the path to the file containing Collector’s service account token. Defaults to /var/run/secrets/kubernetes.io/serviceaccount/token.
    • timeout: Defines how long the Chronosphere Collector waits for a response from the metadata source. Defaults to 1s.
    • refreshInterval: Defines the interval between refreshes if a cache miss didn’t trigger a cache update from Kubelet. Defaults to 10s.

  • metadataToAugment: Adds pod metadata from the associated Kubernetes Pod metadata from which the metrics are sent as labels to the time series. Supported metadata keys are name and namespace.

  • labelsToAugment: Adds pod labels from the associated Kubernetes Pod metadata from which the metrics are sent as labels to the time series. Specify the source Kubernetes label by name, and provide a name for the label as it should appear on the resulting metric time series.

    For example, the default configuration uses the value from the Kubernetes Pod label app as the value for the service_name label on all metrics.

  • annotationsToAugment: Adds annotations from the associated Kubernetes Pod metadata from which the metrics are sent as labels to the time series. Specify the source Kubernetes annotation by name, and provide a name for the label as it should appear on the resulting metric time series.

    For example, the default configuration uses the Kubernetes Pod annotation app.myorg.com/owner value as the value for the team_owner label.

  • cgroupRefreshInterval: Defines how often to refresh the inode to cgroup mapping, if a cache miss didn’t trigger a lookup. Defaults to 5m.

• stripContainerID: Removes the container_id or c label sent by the DogStatsD metrics client. Defaults to true.

Default configuration

The following example shows the default configuration for DogStatsD ingest:

push:
  dogstatsd:
    enabled: true
    listenAddress: 0.0.0.0:8125
    listenProtocol: tcp/udp
    aggregation:
      timers:
        # Default is "histogram" - aggregation to exponential histograms.
        # Valid values are "histogram" and "tdigest".
        # Use `tdigest` only when upgrading from Collector versions prior
        # to v0.110.0 with aggregations explicitly enabled.
        aggregationType: histogram
      interval: 60s
      inactiveExpireAfter: "2m"
    labels:
       env: ${ENV:""}
       k8s_cluster_name: ${KUBERNETES_CLUSTER_NAME:""}
   # prefix: "qa"
   kubernetesMetadataAugmentation:
      enabled: true
      kubelet:
        kubeletNodeIP: ${KUBERNETES_NODE_IP}
        kubeletNodePort: 10250
        # kubeletNodePodsEndpoint: https://${KUBERNETES_NODE_IP}:10250/pods
        kubeletNodeTLSInsecureSkipVerify: true
        kubeletNodeBearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
        refreshInterval: 10s
        timeout: 1s
      metadataToAugment:
        pod: # map of metadata_keyname: label_name
          name: kube_pod_name
          namespace: kube_namespace
      labelsToAugment:
        pod: # map of pod_label: label_name
          app: service_name
      annotationsToAugment:
        pod: # map of pod_annotation: label_name
          "app.myorg.com/owner": owner_team
    stripContainerID: true