Datadog

The Datadog integration requires CXDOT Collector 1.4.0 or greater. Datadog is an observability platform. Use the Datadog integration with the Chronosphere XDOT Collector to collect DogStatsD metrics, Datadog application performance monitoring (APM) traces, and APM span metrics from apps running in your environment.

​

Supported telemetry types

The Datadog integration supports these telemetry types:

​

Prerequisites

The Datadog integration requires the DogStatsD and APM listeners to be reachable from each telemetry sender.

​

Configure

To configure the Datadog integration, follow these steps:

1. Send DogStatsD metrics over UDP port 9125 and APM traffic over TCP port 9126. If you use the admission controller for node-local routing, add the admission.cxdot.paloaltonetworks.com/enabled: "true" label to each pod template that uses Datadog instrumentation.
2. If apps send APM traffic directly to the collector, choose how to produce APM span metrics. Apps that don’t compute APM stats require the collector to compute them. For example, add the following to the values.yaml for your Helm chart:
   config:
     integrations:
       datadog:
         apm:
           stats:
             compute: true
   
   Leave compute set to false when apps compute APM stats or when a Datadog Agent forwards APM traffic. The Agent forwards sampled traces, so computing stats from those traces reports fewer requests than expected. With compute set to false, the collector uses the APM stats payloads from the apps or Agent.
3. Optional: Enable cgroup inode resolution when clients send a cgroup inode instead of a full container ID. For example, add the following to the values.yaml for your Helm chart:
   config:
     integrations:
       datadog:
         origin_detection:
           cgroup_inode: true
   
   This setting mounts the node’s cgroup file system read-only into the collector. Full container IDs don’t require this setting. For information about metadata added after origin detection, see enrichment.
4. Optional: Disable either or both intake capabilities. For example:
   config:
     integrations:
       datadog:
         dogstatsd:
           enabled: false
         apm:
           enabled: false
   
   Set dogstatsd.enabled to false to disable DogStatsD metrics, and set apm.enabled to false to disable APM traces and span metrics.

​

Validate

To validate the Datadog integration, follow these steps:

1. Generate app traffic that emits a DogStatsD metric and an APM trace.
2. In the Live Telemetry Analyzer, filter for cxdot.integration.name=datadog. Confirm that the expected DogStatsD metric names appear. If APM stats are available, also confirm that traces.span.metrics.calls and traces.span.metrics.duration appear.
3. In Metrics Explorer, run the following query:
   count by ("__name__") ({"cxdot.integration.name"="datadog"})
   
   Confirm that the query returns the expected DogStatsD and APM span metric names.
4. In Trace Explorer, filter for cxdot.integration.name=datadog. Confirm that the trace from the generated traffic appears with the expected service name and spans.

​

Troubleshooting

• APM span metrics report fewer requests than expected when a Datadog Agent forwards traffic: Set apm.stats.compute to false so the collector uses the stats payloads that the Agent computes before trace sampling.
• Apps don’t send metrics or traces after manual configuration: Confirm that the apps use port 9125 for DogStatsD and port 9126 for APM, instead of the Datadog Agent defaults. If apps send to the node IP, confirm that the node collector exposes matching hostPort values. Otherwise, send to the node-local collector Service.

For more information about diagnosing a failing integration, see Troubleshooting.

​

Configuration reference

Configure one Datadog integration instance with the following settings. In Helm values, place these settings under config.integrations.datadog. In a Collector configuration file, place them under cxdot.integrations.datadog.

​

Optional settings

• enabled Type: boolean. Optional. Default: true. Whether to enable this Datadog integration instance. If true, the Collector receives telemetry through enabled Datadog intake capabilities. If false, the Collector doesn’t run this integration instance.
• origin_detection Type: object. Optional. Settings for identifying the container that sends DogStatsD metrics or Datadog traces. The Collector uses the container identity to attach workload metadata to the telemetry.
• origin_detection.cgroup_inode Type: boolean. Optional. Default: false. Whether to resolve a cgroup inode sent by a client to its container identity. If true, the Collector mounts the node’s cgroup file system read-only and resolves cgroup inodes. If false, it resolves only full container IDs.
• origin_detection.socket Type: boolean. Optional. Default: false. Whether to identify the sending container from Unix socket credentials. If true, the Collector requires host process access and uses socket credentials for DogStatsD traffic received through a unixgram listener. If false, it doesn’t use socket credentials.
• dogstatsd Type: object. Optional. Settings for receiving and aggregating DogStatsD metrics.
• dogstatsd.enabled Type: boolean. Optional. Default: true. Whether to receive DogStatsD metrics. If true, the Collector listens on the configured DogStatsD addresses. If false, it doesn’t receive DogStatsD metrics.
• dogstatsd.listeners Type: array of object. Optional. Default: [{"endpoint":"0.0.0.0:9125","transport":"udp"}]. Network addresses on which the Collector receives DogStatsD metrics. Configure at least one listener. A unixgram listener requires the node Collector to run as root so that it can create the socket.
• dogstatsd.listeners[].endpoint Type: string. Required. Network address or Unix socket path on which the Collector receives DogStatsD metrics.
• dogstatsd.listeners[].transport Type: string. Required. Transport for the DogStatsD listener. Use udp for a network listener or unixgram for a Unix socket. Allowed values: udp, unixgram.
• dogstatsd.aggregation_interval Type: duration. Optional. Default: 60s. How often the Collector emits aggregated DogStatsD metrics. Set this interval to the licensed metrics resolution.
• dogstatsd.inactive_expire_after Type: duration. Optional. How long an aggregated time series can receive no samples before the Collector expires it and emits a staleness marker. The marker uses the time of the next expected sample, so set this duration within the out-of-order window accepted by the metrics backend.
• apm Type: object. Optional. Settings for receiving Datadog traces and span metrics.
• apm.enabled Type: boolean. Optional. Default: true. Whether to receive Datadog trace traffic. If true, the Collector listens for traces and stats payloads. If false, it doesn’t receive Datadog trace traffic.
• apm.listener Type: object. Optional. Network address on which the Collector receives Datadog trace traffic. A Unix stream socket requires the node Collector to run as root so that it can create the socket.
• apm.listener.endpoint Type: string. Required. Default: 0.0.0.0:9126. Address or Unix socket path on which the Collector receives Datadog trace traffic.
• apm.listener.transport Type: string. Required. Default: tcp. Transport for the trace listener. Use tcp for a network listener or unix for a Unix stream socket. Allowed values: tcp, unix.
• apm.stats Type: object. Optional. Settings for producing span metrics.
• apm.stats.enabled Type: boolean. Optional. Default: true. Whether to produce span metrics. If true, the Collector processes received stats payloads or metrics computed from raw spans. If false, it drops received stats payloads. Set this field to true when compute is true.
• apm.stats.normalize_tags Type: boolean. Optional. Default: true. Whether to normalize span metric label values using Datadog tag character rules. If true, the Collector lowercases label values and replaces unsupported characters. If false, it preserves label values after Datadog stats processing. This setting doesn’t modify traces or span tags.
• apm.stats.compute Type: boolean. Optional. Default: false. Whether the Collector computes span metrics from received traces. If true, the Collector drops received stats payloads and computes metrics from raw spans. If false, it uses stats payloads received from apps or a Datadog Agent. Don’t enable this setting for Agent-forwarded traffic because the Agent forwards sampled traces, which results in a request count lower than the actual count.