Querying DogStatsD formatted metrics

Chronosphere can ingest and use Datadog metrics. The query syntax differs from PromQL, which requires you to build queries differently.

Anatomy of a Datadog query

The following example illustrates the structure of a Datadog query. Other queries might be in a different order:

avg(last_1d):avg:count_nonzero(uptime{app:shopist} by {host}.as_rate()).rollup(avg,3600)<2

This query breaks down into these sections:

• Evaluation window: avg(last_1d)
• Space aggregator: avg
• Function: count_nonzero
• Metric name: uptime
• Filters/scope: app:shopist
• Grouping: host
• Type converter: as_rate()
• Functions: rollup(avg,3600)
• Operators: <2

Visit the Datadog Query Syntax documentation (opens in a new tab) for further information and examples.

Query syntax and modes

Querying DogStatsD metrics in Chronosphere are based on the different modes set in the Collector.

Metrics storage in the backend depends on the mode configured in the dogstatsd section of the push configuration in the Collector.

The query syntax is slightly different for each mode.

• regular

  The DogStatsD METRIC_NAME maps to the Prometheus __name__ label, replacing all non-alphanumeric and non-dot characters with underscores. Dots convert to an underscore ( _ ). Any labels defined on the metric remain unchanged and append to the list of labels. Refer to Prometheus naming recommendations for specific information.

• graphite

  The Prometheus __name__ label gets a constant stat name and the DogStatsD METRIC_NAME assigns to a Prometheus label set in the configuration namelabelname (by default name).

• graphite_expanded

  The expanded Graphite mode is the same as graphite mode, except in addition to storing everything in the namelabelname label, the METRIC_NAME separates on dot ( . ) and stores each part in a separate label. For example, t0, t1, and t2.

Here's an example of a DogStatsD query:

users.online:2\|c\|#country:france

The following table shows examples of the same query in each of the Collector mode configurations:

Querying best practices

For graphite_expanded metrics, it's best to start your query with stat, and then search for either t0 or the defined labels using autocomplete. By starting with stat, your search scope focuses on the DogStatsD metrics, which improves query performance.

For example, using the previous metric (users.online:2|c|#country:france), you can start your query with stat, add t0 and using autocomplete, and search for users. Then, search for t1 and so on.

Metric types and querying

All metrics convert to Prometheus metric types before storage in Chronosphere. Most metric types are the same across DogStatsD and Prometheus with the exception of counters.

Counters in Prometheus are running counters, which means they always increase or remain constant, and never decrease. DogStatsD counters are DELTA counters. When querying counters in Chronosphere, apply a rate () function (opens in a new tab).