> ## Documentation Index > Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt > Use this file to discover all available pages before exploring further. # Case study: protecting a telemetry data budget > A narrative walkthrough of an administrator investigating a telemetry data spike, repartitioning consumption, and revising budgets in Chronosphere Observability Platform. A license protects nothing on its own. This case study follows an administrator who owns an observability budget through a single morning, from a budget threshold alert to a partition hierarchy and a set of budgets that hold. The case study shows how the control tools in Chronosphere Observability Platform connect into one workflow for finding a spike, attributing it to an owner, and using shaping tools to reshape the data. For a step-by-step reference version of this case study, see the [Manage telemetry data budgets guide](/overview/guides/manage-budgets). ## The threshold breach The Slack channel lit up at 9:42 AM, and it wasn't an incident. > **WARNING: Log persisted bytes exceeded the daily volume warning threshold** > > *Budget: commerce-platform-logs* > *Partition: global/commerce-platform* > *Threshold: 1.6 TB daily volume, LOG\_PERSISTED\_BYTES* Nadia Vance read it twice. Daily budget windows reset at midnight UTC, and she worked in the UTC time zone, so the reset landed at her own midnight. She had written that budget herself six weeks earlier, which included a warning at 1.6 TB, a critical alert at 1.9 TB, and a drop action at 2 TB of persisted log bytes per day. The Commerce Platform team had never crossed 1 TB. Now, nine hours into the UTC day, the partition had persisted 1.61 TB, and the day wasn't close to over. The alert had reached her because budget alerts behave like any other alert in Observability Platform. Her daily volume thresholds had generated a monitor in the Chronosphere Managed [collection](/administer/collections), one for the threshold type rather than one for each alert. The `budget-alerts-slack` [notification policy](/investigate/alerts/notifications/policies) attached to the budget routed the alert to the channel she watched. The annotations she had added to `alert_action_config` included a link to her own runbook. The clock mattered. If consumption kept its current rate, the 2 TB drop threshold would trip shortly after noon, and Observability Platform would then drop Commerce Platform log data until the daily window reset at midnight UTC. Production logs would go with the rest, because her budget had no priorities to say otherwise. ## Checking the contract clock Before targeting a single partition, Nadia wanted to know how much license usage remained for the organization. She opened **License Overview** and stayed on the **Snapshot** tab, where the [**Credit consumption**](/administer/limits-licensing/licensing#credits-consumption) section compares spend against the contract period. Two numbers sat next to each other: * **Credit consumption (full contract)**: 63% * **Elapsed time in contract**: 47% Her organization was spending credits faster than the contract clock was running. The **Credits cumulative trend** chart showed the divergence starting a week ago, which matched nothing she had approved. Credits are fungible across telemetry types, so a log spike in one team spends the same pool that metrics and traces draw from everywhere else. This wasn't a Commerce Platform problem she could postpone. Nadia clicked **Go to Consumption** to view the larger trends. ## Where the growth lives The [**Consumption**](/control/consumption/analyze) page opened for the global partition. Nadia set the **Time window** menu to **Daily** and read the shape of the last two weeks. The Commerce Platform partition held a flat line near 950 GB per day, then stepped up to a new plateau trending toward 4 TB. Switching the window to **Per second** provided a clearer view. Consumption for `global/commerce-platform` had jumped from 11 MB per second to 46 MB per second at 3:12 AM UTC and stayed there. Spikes taper, but this was a step function, which meant a configuration change rather than a traffic surge. Setting **Group by** to **Consumed by partition** ranked the offenders. Commerce Platform accounted for 61% of global consumption, more than the next three partitions combined. In the **Child Partitions** section, Nadia drilled in: | Child partition | Share of team consumption | Two weeks ago | | ------------------ | ------------------------- | ------------- | | `checkout-service` | 74% | 21% | | `payment-gateway` | 14% | 38% | | `order-service` | 9% | 33% | | `default` | 3% | 8% | The **Budgets** panel reported **Alerting** for the team partition, and the **Current budget status** column showed no badge for `checkout-service`, because that child partition had no budget attached. Nadia had partitioned Commerce Platform by service and then stopped, so the only threshold in the hierarchy sat one level up from the service causing the problem. Whatever `checkout-service` did next, the team partition absorbed it. ## Which resource is growing Consumption is tracked for each resource rather than for telemetry data as a whole, so Nadia checked which resources had moved. Over the same window: * `LOG_PERSISTED_BYTES` for `checkout-service` was up 15 times. * `LOG_PROCESSED_BYTES` rose almost in step, because the parsers and control rules matched the new volume for transformation. * `TRACE_PERSISTED_BYTES` was up 18%, a smaller but real increase. * Metric resources were flat. That distinction changed what Nadia could do about each resource. Log resource groups accept both alert and drop actions, so logs could be capped. Trace resource groups don't support the `DROP` action, so the trace increase would need [sampling](/control/shaping/sample-traces) rather than a threshold. For the definitions behind each resource, Nadia kept [Consumption licensing concepts](/administer/limits-licensing/concepts/consumption-licensing) open in a second tab. ## The pattern behind the bytes Volume told her which service to focus on, but it didn't tell her which logs. Nadia opened the [**Logs Usage**](/investigate/analyze/logs-usage) page, which groups log data into patterns of frequent `service` and `message` combinations, reports the volume each pattern consumes, and scores how often anyone queries it. Sorted by volume, the top pattern was unfamiliar: | Pattern | Daily volume | Utility score | | ------------------------------------------------------ | ------------ | ------------- | | `checkout-service` / `payment retry attempt scheduled` | 2.8 TB | 2 | | `checkout-service` / `order submitted` | 41 GB | 46 | | `payment-gateway` / `charge authorized` | 38 GB | 51 | Utility scores are relative rather than scaled, so the comparison is what's important. Looking at the data, a pattern consuming 68 times the bytes of `order submitted` was queried by almost nobody. On the **Pattern Details** page, the **Executions** column named a single user, who had run one query against the pattern four days earlier. Nadia held the pointer over the pattern and clicked **View in Logs Explorer** to further explore the underlying data. ## Confirming it in the logs [**Logs Explorer**](/investigate/querying/query-logs) opened with the pattern's filter already in the query box. Nadia widened it to every debug log `checkout-service` emitted and grouped the results to see where they came from by running the following query: ```kql theme={null} service = "checkout-service" AND severity = "DEBUG" | summarize total = count() by message, environment | sort by total desc | limit 10 ``` The table answered two questions at once. The retry message dominated every other debug log, and 71% of those logs came from the `staging` and `dev` environments. Only 29% came from `production`. She expanded one log: ```text theme={null} 2026-08-17T03:12:44Z [DEBUG] checkout-service/payment_client.go:214 payment retry attempt scheduled backoff=250ms attempt=1 provider=payment-api.prod.internal environment=staging ``` The timestamp matched the step on the **Consumption** page exactly. To find what shipped at that moment, Nadia opened **Changes Explorer**, ran the **Code deploys** quick filter, and narrowed the window to the minutes around 3:10 AM UTC. One [change event](/observe/enable-events/use-events) accounted for the step, and the labels on its **Summary** tab named the source: > **Deploy: checkout-service v3.42.0** > *source: ArgoCD* > *deployed-by: deploy-bot* Then she recognized it. Three weeks earlier, an on-call engineer had traced a production outage to a DNS misconfiguration in `v3.41.0` and rolled it back, a sequence recorded in the [incident response case study](/overview/case-studies/respond-to-incidents). The debug logging that engineering enabled during that postmortem had shipped in the next release and had never been turned off. A verbose logger for one payment client was now consuming more than two thirds of a team's daily log volume. Back in Logs Explorer, Nadia selected the **Time chart** visualization and clicked **Add to notebook**, naming the [notebook](/navigate/notebooks) `Checkout log budget breach`. Observability Platform added two panels, a log volume histogram and a time chart of her query, and she added the log line as a note in a record she would share later. ## A finer partition Fixing the logger was engineering's work, and it would take a release cycle. Nadia's job was to ensure the next verbose logger cost less. That meant partitioning `checkout-service` finely enough that a threshold could target the part of it no one depended on. She added environment child partitions to the existing service partition, giving `prod`, `staging`, and `dev` a place of their own in the hierarchy. ```mermaid actions={false} theme={null} --- title: Checkout service partitions after the change --- flowchart TD accTitle: The checkout-service partition with new environment child partitions accDescr: The global partition contains a commerce-platform partition, which contains checkout-service, payment-gateway, and order-service partitions plus an automatic default partition. The checkout-service partition now has child partitions for prod, staging, dev, and its own automatic default partition. global[global] --> commerce[commerce-platform] commerce --> checkout[checkout-service] commerce --> payments[payment-gateway] commerce --> orders[order-service] commerce --> commerce_default[default] checkout --> prod[prod] checkout --> staging[staging] checkout --> dev[dev] checkout --> checkout_default[default] classDef green fill:#28a561,stroke-width:2px,stroke:#28a561,color:#FFFFFF,font-weight:bold; classDef orange fill:#ff9d1c,stroke-width:2px,stroke:#ff9d1c,color:#FFFFFF,font-weight:bold; class prod,staging,dev green class checkout_default,commerce_default orange ``` Three details shaped how she wrote the definition. Partitions support a maximum nesting depth of five levels from `global`, so adding an environment level under `checkout-service` still left her room to subdivide later. Observability Platform creates a default partition automatically at every level, so any checkout traffic that matched no environment filter would still sum into the service total rather than disappearing. The slug becomes permanent when the configuration is applied, so she settled the naming before she touched anything. Nadia also added a `NOT_IN` filter to the service partition to keep load-test traffic out of the team's totals. That traffic had been quietly inflating the baseline she set her original threshold against. The following definition shows the `checkout-service` branch of the hierarchy as it sits inside the `commerce-platform` partition, with the three new environment child partitions: ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: ConsumptionConfig spec: partitions: - name: "Checkout Service" slug: "checkout-service" filters: - operator: "IN" conditions: - log_filter: query: "service = 'checkout-service'" - metric_filters: - name: "service" value_glob: "checkout-service" - trace_span_filters: - service: match: "EXACT" value: "checkout-service" - operator: "NOT_IN" conditions: - log_filter: query: "environment = 'load-test'" - metric_filters: - name: "environment" value_glob: "load-test" partitions: - name: "Prod" slug: "prod" filters: - operator: "IN" conditions: - log_filter: query: "environment = 'production'" - name: "Staging" slug: "staging" filters: - operator: "IN" conditions: - log_filter: query: "environment = 'staging'" - name: "Dev" slug: "dev" filters: - operator: "IN" conditions: - log_filter: query: "environment = 'dev'" ``` Because partitions apply in match order and the first match wins, Nadia placed the environment partitions ahead of any broad catch-all in the configuration file. For the full mechanics of filters, operators, and applying the definition, see [Partitions](/control/consumption/partitions#create-partitions). She applied the change with [Chronoctl](/tooling/chronoctl), the same way the rest of her consumption configuration was managed. ## Rewriting the budget New partitions gave Nadia somewhere to enforce a threshold. She attached a [budget](/control/consumption/budgeting) to the `staging` partition with an instant-rate threshold, which limits the rate of incoming data in bytes per second, and reacts within seconds rather than waiting on a cumulative window. ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: ConsumptionBudget spec: slug: "checkout-staging-logs" name: "Checkout Staging Logs" partition_slug_path: "global/commerce-platform/checkout-service/staging" alert_action_config: annotations: runbook: "https://wiki.example.com/runbooks/log-budget-alerts" description: "Checkout staging log rate exceeded its budget" labels: team: "commerce-platform" environment: "staging" instant_rate_sustain_secs: 300 thresholds: - action: ALERT_WARN instant_rate: fixed_value_per_sec: 5000000 type: INSTANT_RATE resource_group: LOG_PERSISTED_BYTES unit: NATIVE - action: DROP instant_rate: fixed_value_per_sec: 6000000 type: INSTANT_RATE resource_group: LOG_PERSISTED_BYTES unit: NATIVE notification_policy_slug: "budget-alerts-slack" ``` The `instant_rate_sustain_secs: 300` setting was the difference between an alert worth reading and a pager she would learn to ignore. The default of `0` triggers on any breach, and a staging environment breaches for a few seconds regularly. Five minutes of sustained overage, however, meant something had actually changed. She duplicated the budget for `dev` at a lower rate, and then noticed a second benefit of the two new budgets. Budgets are evaluated from the most specific partition upward, and data a child budget drops never counts against a parent's received rate. A 6 MB per second cap on staging meant the team partition would see at most 6 MB per second from staging no matter what engineering deployed next. The offending environment would absorb its own drops. For an example of that evaluation order, see [Budgeting order](/control/consumption/budgeting#budgeting-order). Then she revised the team budget that had paged her. The three changes she made were as follows: * A `ROLLING_1_HOUR_VOLUME` warning threshold came first. Her daily threshold had told her at 9:42 AM about a change that began at 3:12 AM. A one-hour rolling window would have told her by 4:15 AM. * Priorities stopped a drop action from being indiscriminate: `environment = 'dev'` at `10`, `environment = 'staging'` at `5`, and `environment = 'production'` at `1`. Higher values drop first, so production logs now drop last. She set `default_priority: 5` so unmatched traffic wouldn't inherit the default of `10` and vanish first. * An `ALERT_WARN` threshold on `TRACE_PERSISTED_BYTES` with a `ROLLING_1_DAY_VOLUME` window covered the traces. Trace resource groups reject the `DROP` action, so visibility was the only threshold she could set there. ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: ConsumptionBudget spec: slug: "commerce-platform-logs" name: "Commerce Platform Logs" partition_slug_path: "global/commerce-platform" alert_action_config: annotations: runbook: "https://wiki.example.com/runbooks/log-budget-alerts" description: "Commerce Platform consumption exceeded a budget threshold" labels: team: "commerce-platform" default_priority: 5 priorities: - filters: - log_filter: query: "environment = 'dev'" priority: 10 - filters: - log_filter: query: "environment = 'staging'" priority: 5 - filters: - log_filter: query: "environment = 'production'" priority: 1 thresholds: - action: ALERT_WARN volume: fixed_value: 90000000000 type: ROLLING_1_HOUR_VOLUME resource_group: LOG_PERSISTED_BYTES unit: NATIVE - action: ALERT_WARN volume: fixed_value: 1600000000000 type: DAILY_VOLUME resource_group: LOG_PERSISTED_BYTES unit: NATIVE - action: ALERT_CRITICAL volume: fixed_value: 1900000000000 type: DAILY_VOLUME resource_group: LOG_PERSISTED_BYTES unit: NATIVE - action: DROP volume: fixed_value: 2000000000000 type: DAILY_VOLUME resource_group: LOG_PERSISTED_BYTES unit: NATIVE - action: ALERT_WARN volume: fixed_value: 400000000000 type: ROLLING_1_DAY_VOLUME resource_group: TRACE_PERSISTED_BYTES unit: NATIVE ``` She resisted the temptation to add weekly and monthly drop thresholds alongside the daily one. The windows overlap enough that the same growth trips all three and pages her three times for one problem. For the available windows and the rules on which resource groups accept which type, see [Threshold types](/control/consumption/budgeting#threshold-types). To change a threshold later, see [Update budgets](/control/consumption/budgeting#update-budgets). Removing a threshold that carries an alert action also deletes the monitor generated from it. ## Shaping instead of dropping The thresholds were a backstop. They drop data only after consumption has already run past a limit, and priorities control the order of those drops rather than the value of the data inside each priority. To bring consumption back under the license limit, Nadia needed shaping rules. She returned to the pattern on the **Logs Usage** page and clicked **Create control rule for this pattern**, which opened the [control rule](/control/shaping/shape-logs/control-logs) editor with the pattern's filter already in place. The editor previewed the logs that filter matched, so she could confirm it caught the retry pattern and nothing adjacent. The obvious rule was to drop the retry logs. Before writing it, she messaged the engineer who had queried the pattern, and learned that the retry count itself was worth keeping, even though nobody wanted the log lines. Nadia chose **Emit metric** rather than **Drop logs**. The rule converted each matching log into an increment on a counter named `checkout_payment_retry_total`, labeled with `service` and `environment`, and she selected **Drop original log after emitting metric**. A 2.8 TB per day log pattern became a metric counter, and the signal the engineer cared about survived the reduction. She confirmed the replacement in [**Metrics Explorer**](/investigate/querying/metrics/explorer): ```text wrap theme={null} sum(rate(checkout_payment_retry_total{environment="production"}[5m])) by (service) ``` The counter was populating. Retry behavior was still observable, and the bytes were gone. For the other levers each telemetry type offers, including **Drop fields** and **Sample logs** for logs, drop rules for metrics, and sampling for traces, see [Shaping rules](/control/shaping). The trace increase needed its own approach, because no threshold can drop trace bytes. She created a [trace dataset](/control/shaping/sample-traces/datasets) for `checkout-service` and applied a [behavior](/control/shaping/sample-traces/behaviors) that lowered the sampling rate for non-production spans, previewing the behavior against the dataset before assigning it. ## Proof the changes held By 2:30 PM, Nadia wanted evidence rather than intent. She looked for three things, and got the last of them the following morning. On the **Consumption** page, the **Per second** window for the team partition had fallen from 46 MB per second to 13 MB per second, a daily rate of 1.1 TB rather than 4 TB. The current daily window had already spent its budget by then, so the **Budgets** panel didn't clear its status badge until the window reset at midnight UTC. The 2 TB drop threshold had tripped a few minutes past noon, as she expected at 9:42 AM, but the priorities she added in the meantime decided what went. She set **Group by** to **Consumed vs dropped** to see the shape of the drops. The only dropped data was in priority `10` and priority `5`, in the staging and development partitions, exactly where her priorities said it should be. Production had dropped nothing. The next morning, the retry pattern on the **Logs Usage** page reported 24 GB instead of 2.8 TB. That page is where a log control rule's impact shows up, as described in [Review shaping rules impact](/control/shaping/review-impact). The remainder came from retry messages whose text varied enough to fall outside the rule's filter. Back on the generated monitor, she left a [comment](/navigate/comments) for the Commerce Platform team: > "Your daily log budget breach came from `LOG_LEVEL=debug` shipping in > checkout-service v3.42.0, left over from the v3.41.0 postmortem. The retry logs are > now an emitted metric, `checkout_payment_retry_total`. Staging and development have > their own partitions and instant-rate budgets, so a verbose logger outside production > won't spend the team's production allowance again. Turn the log level back down in > the next release anyway." Then she shared the notebook, which held the two **Logs Explorer** panels along with her notes on the log line, the deploy, the pattern volume before and after, and the new threshold values. ## After The credit trend on the **Snapshot** tab took two days to bend. Credits already spent don't regenerate, so cumulative consumption stayed at 63% against 47% of elapsed contract time. What changed was the slope: the projected spend for the remainder of the contract fell back within the license, and the gap stopped widening. What changed permanently was the shape of the hierarchy. Commerce Platform had gone from a single team-level threshold to per-environment partitions with their own budgets, which meant the next verbose logger in staging would be capped by staging. Nadia put a recurring review on her calendar to rank partitions on the **Consumption** page, and tighten alert thresholds where consumption had become predictable. She would add child partitions wherever a team-level threshold was the only thing between one service and the license. Engineering added a deployment pipeline step that fails any release build with debug logging enabled. That step cost nothing and would have saved 2.8 TB a day. The license would still not defend itself, but the environments that had caused the breach now had thresholds of their own.