Manage telemetry data budgets

Telemetry data that grows unchecked consumes license capacity before the most valuable data arrives. This guide walks through the end-to-end workflow for administrators who own the observability budget in Chronosphere Observability Platform. You’ll learn about partitioning data by owner, analyzing what each partition consumes, setting thresholds that alert or drop data, and shaping telemetry so consumption fits the license.

​

Step 1: Partition your telemetry data

Budgets attach to partitions, so start by dividing consumption into partitions that each have an owner who can act on them. Partitions are hierarchical. Every partition is a child of the global partition, which captures all consumption. Observability Platform adds a default partition at each level to collect traffic that no sibling partition matches, so consumption always sums correctly into the parent partition.

​

Model partitions on ownership

Partition by the boundaries your organization uses to assign responsibility, such as team, service, and environment. A team-level partition shows which group is consuming the license, and child partitions for individual services and environments show which workload inside that team is responsible. Granular child partitions also give you a place to enforce a threshold without penalizing the parent partition. For a diagram of a team, service, and environment hierarchy, see Partition hierarchy.

​

Match data with filters

Each partition uses filters to claim telemetry data, and a request must match every filter to be attributed to that partition. The IN operator requires a request to match at least one of its conditions, and the NOT_IN operator requires a request to match none of them. Use NOT_IN to keep traffic such as load tests out of a team’s consumption totals. Write one condition for each telemetry type you want the partition to capture:

• log_filter matches log data with a log query.
• metric_filters matches metric data by label, and label values accept glob patterns such as service:{svc1,svc2}.
• trace_span_filters matches trace data on span fields such as service, operation, duration, and error.

Partitions apply in the match order listed in your configuration, and the first match wins. Order specific partitions before broad ones so a catch-all filter doesn’t absorb data that belongs to a service partition.

​

Create a partition

To create partitions in Observability Platform, along with the Chronoctl and Terraform equivalents, see Create partitions. Settle the naming scheme before you apply the configuration, because applying it sets the partition’s slug, which can’t be changed afterward.

​

Step 2: Analyze consumption

Measure consumption before you enforce drops. The Consumption page compares the data you persist against your license capacity. Use that comparison to choose a threshold value high enough to protect the data your teams depend on. When you’re already viewing a partition on the Partitions and Budgets page, click View in Consumption to open the Consumption page scoped to that partition.

​

Find what’s driving growth

In the Consumption panel, use the Time window menu to match the period to the pattern you’re looking for. The Per second window exposes spikes as they happen. The Daily, Weekly, and Monthly (Cumulative) windows expose gradual growth and sustained overconsumption. The rolling windows, from Rolling 30 minutes through Rolling 7 days, expose anomalies that recur over a few hours or days. Set the Group by menu to Consumed by partition to rank partitions against each other. The partition at the top of that ranking is where a budget produces the most benefit. Use the Child Partitions section to drill into a team partition and find the specific service or environment responsible.

​

Know which resources you consume

Consumption is tracked for each resource, not for telemetry data as a whole, and budget thresholds apply to a single resource group. Logs and traces consume persisted bytes for data stored in the database and processed bytes for data that shaping rules match for transformation. Metrics consume persisted data points, persisted series, and matched data points. Identify which of those resources is growing before you choose a threshold, because a partition can stay well within its persisted bytes threshold while its processed bytes increase. For definitions of each resource, see Consumption licensing concepts.

​

Step 3: Create budgets and set thresholds

A budget is a shaping policy attached to a partition. Budgets hold the thresholds that alert or drop data, along with the priorities that decide which data is dropped first. Before you create one, define the notification policy that routes budget alerts, and note the slug path of the target partition.

​

Choose a threshold type

Match the threshold type to the behavior you’re defending against. Instant-rate and rolling thresholds use short windows, so they react quickly to spikes and generate more noise. Fixed volume thresholds reset daily, weekly, or monthly, so they catch gradual growth with fewer false signals. Instant-rate thresholds apply only to the LOG_PERSISTED_BYTES and LOG_PROCESSED_BYTES resource groups. For metric, trace, and credit-based resource groups, use a rolling or fixed volume threshold. For the full list of window lengths, see Threshold types. Each combination of resource group, action, and threshold type must be unique within a budget. The global partition already enforces the rate limits in your contract as system thresholds, so set custom thresholds on child partitions instead.

​

Alert before you drop

Configure alert actions first, and add drop actions after you’ve watched the alerts for a full window. Alerts show what a threshold value would have dropped without actually dropping data. Set the warning alert at a lower value than the point where you’d drop data, and set the critical alert at that point. Each alert action generates a monitor in the Chronosphere Managed collection, but a budget produces only one monitor for each threshold type. Several alerts on the same threshold type share a single monitor. In the alert_action_config section of the budget definition, add the metadata that the responder needs when the monitor pages them:

• annotations for runbook links and a description that explains which partition is over budget.
• labels such as team and environment, which notification policies use to route the alert to the group that owns the data.
• instant_rate_sustain_secs to require that consumption exceed an instant-rate threshold for a set duration before the alert triggers. The default of 0 triggers on any breach.

​

Decide what gets dropped first

Priorities control drop order within a budget. Higher values are dropped first, so data with a priority of 10 is dropped before data with a priority of 1. Assign a high priority value to your lowest-value traffic, such as development and staging environments, and protect production with a low value. Priorities evaluate in match order, and the first match applies. Set default_priority to control traffic that matches no priority filter. Without it, unmatched traffic defaults to 10 and is dropped first. The DROP action isn’t supported for the TRACE_PROCESSED_BYTES, TRACE_PERSISTED_BYTES, TRACE_ALL, or cross-telemetry ALL resource groups. Use an alert action for those resource groups, and reduce trace volume with sampling instead.

​

Attach the budget to a partition

The threshold type, action, and priorities described in this section correspond to the fields you fill in when you create the budget. For complete definitions that include thresholds, priorities, and alert metadata, see the Chronoctl and Terraform budget examples.

​

Understand evaluation order

When budgets exist at several levels of the hierarchy, a request passes through each one, starting with the most specific partition and ending at the global partition. Data that a budget drops never reaches the received rate of its parent budgets. A child budget therefore protects its parents. If a service partition with a 1 MB per second instant-rate threshold receives a 1 GB per second spike, its parent team partition still sees only 1 MB per second. The offending service absorbs the drops, and the rest of the team’s data continues to flow. For an example, see Budgeting order.

​

Step 4: Monitor budgets

A budget is a hypothesis about future consumption. Watch how it behaves so you can correct the threshold before it drops data you need.

​

Track budget status

On the Consumption page, the Budgets panel shows a status badge of Ok, Alerting, or Dropping for the selected partition. Partitions with no budget display no badge. The panel also lists each threshold and how much of it the partition has consumed. Where you configured thresholds, a dotted line marks the threshold value on the consumption graph, which makes headroom visible at a glance.

• Select Consumed vs dropped in the Group by menu to see the rate of data that thresholds dropped for the selected partition, displayed by priority.
• Compare a partition’s trend against its threshold line across several time windows to distinguish a one-time spike from sustained growth.
• Review the Child Partitions section to find which child is responsible when a parent partition alerts. The Current budget status column reports the status of each child partition’s budget.

The global partition budget can exceed its threshold while every child budget stays within its own. Setting thresholds on individual partitions makes the global status reflect real growth trends instead of aggregate noise.

​

Triage a budget alert

Budget alerts behave like any other alert in Observability Platform, so they appear on the Alerts page and route through the notification policy attached to the budget. The generated monitor carries the annotations you defined, so a runbook link reaches an on-call engineer at the moment they need it. For the triage guide, see Respond to incidents. When a budget starts dropping data, choose a response based on what the consumption data shows:

• A short spike from a known cause, such as a load test: Exclude that traffic from the partition with a NOT_IN filter.
• Sustained growth in low-value data: Keep the threshold and shape the data so consumption fits the budget.
• Sustained growth in data teams depend on: Raise the threshold and lower the thresholds on sibling partitions so total consumption still fits your license.

Removing a threshold that has an alert action deletes the monitor generated from it, along with the alert history attached to that monitor. Updates to notification_policy or alert_action_config propagate to the generated monitor automatically. To change or remove thresholds, see Update budgets.

​

Step 5: Shape telemetry data

Thresholds cap consumption by dropping data indiscriminately within a priority. Shaping rules reduce consumption by removing or condensing data that delivers little value, which keeps you inside the budget without losing signal.

​

Reduce log volume

Shape log data in the order Observability Platform applies the rules. Parse plaintext logs into structured fields, normalize field names, and then create control rules to drop whole logs, drop individual fields, or sample a percentage of a log stream. Dropping a verbose field across a high-volume service often recovers more persisted bytes than dropping an entire log source.

​

Reduce metric volume and cardinality

Shape metric data with rules that match how the metric is used. Drop rules remove metrics that nothing queries, aggregation and rollup rules condense high-cardinality series into the dimensions teams query, and recording rules calculate expensive expressions in advance. For a workflow that identifies the offending labels first, see Reduce metric cardinality.

​

Sample trace data

Because trace resource groups don’t support drop actions, sampling is how you control trace consumption. Create trace datasets that map trace volume to business units, then apply behaviors to set sampling rates for those datasets without writing individual head and tail sampling rules.

​

Confirm the rules worked

After a shaping rule takes effect, review rule impact to verify the volume reduction you expected, and return to the Consumption page to confirm the partition now stays within its threshold.

​

What to do after budgets are in place

Budget management is a recurring review rather than a one-time configuration. Revisit the following as consumption changes:

• Refine partitions: When a team partition consistently drives growth, add child partitions for its services or environments so a threshold can target the specific workload responsible.
• Tighten thresholds: After consumption becomes predictable, lower alert thresholds so they warn earlier, and keep drop thresholds close to the values your license supports.
• Improve generated monitors: Add or update annotations in alert_action_config so the next responder gets a runbook link and a description that names the partition.
• Review your license: Compare thresholds against the resources and limits in Licensing so thresholds reflect what your contract covers and the penalties for exceeding it.
• Revise accordingly: A partition with an attached budget can’t be deleted, so delete the budget before you delete the partition.