The threshold breach
The Slack channel lit up at 9:42 AM, and it wasn’t an incident.WARNING: Log persisted bytes exceeded the daily volume warning threshold Budget: commerce-platform-logs Partition: global/commerce-platform Threshold: 1.6 TB daily volume, LOG_PERSISTED_BYTESNadia Vance read it twice. Daily budget windows reset at midnight UTC, and she worked in the UTC time zone, so the reset landed at her own midnight. She had written that budget herself six weeks earlier, which included a warning at 1.6 TB, a critical alert at 1.9 TB, and a drop action at 2 TB of persisted log bytes per day. The Commerce Platform team had never crossed 1 TB. Now, nine hours into the UTC day, the partition had persisted 1.61 TB, and the day wasn’t close to over. The alert had reached her because budget alerts behave like any other alert in Observability Platform. Her daily volume thresholds had generated a monitor in the Chronosphere Managed collection, one for the threshold type rather than one for each alert. The
budget-alerts-slack
notification policy attached to the
budget routed the alert to the channel she watched. The annotations she had added to
alert_action_config included a link to her own runbook.
The clock mattered. If consumption kept its current rate, the 2 TB drop
threshold would trip shortly after noon, and Observability Platform would then
drop Commerce Platform log data until the daily window reset at midnight UTC.
Production logs would go with the rest, because her budget had no priorities to say
otherwise.
Checking the contract clock
Before targeting a single partition, Nadia wanted to know how much license usage remained for the organization. She opened License Overview and stayed on the Snapshot tab, where the Credit consumption section compares spend against the contract period. Two numbers sat next to each other:- Credit consumption (full contract): 63%
- Elapsed time in contract: 47%
Where the growth lives
The Consumption page opened for the global partition. Nadia set the Time window menu to Daily and read the shape of the last two weeks. The Commerce Platform partition held a flat line near 950 GB per day, then stepped up to a new plateau trending toward 4 TB. Switching the window to Per second provided a clearer view. Consumption forglobal/commerce-platform had jumped from 11 MB per second to 46 MB per
second at 3:12 AM UTC and stayed there. Spikes taper, but this was a step function,
which meant a configuration change rather than a traffic surge.
Setting Group by to Consumed by partition ranked the offenders. Commerce
Platform accounted for 61% of global consumption, more than the next three partitions
combined. In the Child Partitions section, Nadia drilled in:
The Budgets panel reported Alerting for the team partition, and the
Current budget status column showed no badge for
checkout-service, because that
child partition had no budget attached. Nadia had partitioned Commerce Platform by
service and then stopped, so the only threshold in the hierarchy sat one level up from
the service causing the problem. Whatever checkout-service did next, the team
partition absorbed it.
Which resource is growing
Consumption is tracked for each resource rather than for telemetry data as a whole, so Nadia checked which resources had moved. Over the same window:LOG_PERSISTED_BYTESforcheckout-servicewas up 15 times.LOG_PROCESSED_BYTESrose almost in step, because the parsers and control rules matched the new volume for transformation.TRACE_PERSISTED_BYTESwas up 18%, a smaller but real increase.- Metric resources were flat.
DROP action, so the trace increase would need
sampling rather than a threshold. For the
definitions behind each resource, Nadia kept
Consumption licensing concepts
open in a second tab.
The pattern behind the bytes
Volume told her which service to focus on, but it didn’t tell her which logs. Nadia opened the Logs Usage page, which groups log data into patterns of frequentservice and message combinations, reports the volume
each pattern consumes, and scores how often anyone queries it.
Sorted by volume, the top pattern was unfamiliar:
Utility scores are relative rather than scaled, so the comparison is what’s
important. Looking at the data, a pattern consuming 68 times the bytes of
order submitted was queried by almost nobody. On the Pattern Details page, the
Executions column named a single user, who had run one query against the pattern
four days earlier.
Nadia held the pointer over the pattern and clicked View in Logs Explorer to
further explore the underlying data.
Confirming it in the logs
Logs Explorer opened with the pattern’s filter already in the query box. Nadia widened it to every debug logcheckout-service emitted and grouped the results to see where they came from by
running the following query:
staging and dev environments. Only
29% came from production. She expanded one log:
Deploy: checkout-service v3.42.0 source: ArgoCD deployed-by: deploy-botThen she recognized it. Three weeks earlier, an on-call engineer had traced a production outage to a DNS misconfiguration in
v3.41.0 and rolled it back, a
sequence recorded in the
incident response case study. The
debug logging that engineering enabled during that postmortem had shipped in the next
release and had never been turned off. A verbose logger for one payment client was now
consuming more than two thirds of a team’s daily log volume.
Back in Logs Explorer, Nadia selected the Time chart visualization and clicked
Add to notebook, naming the notebook
Checkout log budget breach. Observability Platform added two panels, a log volume
histogram and a time chart of her query, and she added the log line as a note in a
record she would share later.
A finer partition
Fixing the logger was engineering’s work, and it would take a release cycle. Nadia’s job was to ensure the next verbose logger cost less. That meant partitioningcheckout-service finely enough that a threshold could target the part of it no one
depended on.
She added environment child partitions to the existing service partition, giving
prod, staging, and dev a place of their own in the hierarchy.
Three details shaped how she wrote the definition. Partitions support a maximum nesting
depth of five levels from global, so adding an environment level under
checkout-service still left her room to subdivide later. Observability Platform creates
a default partition automatically at every level, so any checkout traffic that matched no
environment filter would still sum into the service total rather than disappearing. The
slug becomes permanent when the configuration is applied, so she settled the naming
before she touched anything.
Nadia also added a NOT_IN filter to the service partition to keep load-test traffic
out of the team’s totals. That traffic had been quietly inflating the baseline she set
her original threshold against.
The following definition shows the checkout-service branch of the hierarchy as it
sits inside the commerce-platform partition, with the three new environment child
partitions:
Chronoctl example
Rewriting the budget
New partitions gave Nadia somewhere to enforce a threshold. She attached a budget to thestaging partition with an
instant-rate threshold, which limits the rate of incoming data in bytes per second,
and reacts within seconds rather than waiting on a cumulative window.
Chronoctl example
instant_rate_sustain_secs: 300 setting was the difference between an alert worth
reading and a pager she would learn to ignore. The default of 0 triggers on any breach, and
a staging environment breaches for a few seconds regularly. Five minutes of
sustained overage, however, meant something had actually changed.
She duplicated the budget for dev at a lower rate, and then noticed a second benefit
of the two new budgets. Budgets are evaluated from the most specific
partition upward, and data a child budget drops never counts against a parent’s
received rate. A 6 MB per second cap on staging meant the team partition would
see at most 6 MB per second from staging no matter what engineering deployed
next. The offending environment would absorb its own drops. For an example of
that evaluation order, see
Budgeting order.
Then she revised the team budget that had paged her. The three changes she made were
as follows:
- A
ROLLING_1_HOUR_VOLUMEwarning threshold came first. Her daily threshold had told her at 9:42 AM about a change that began at 3:12 AM. A one-hour rolling window would have told her by 4:15 AM. - Priorities stopped a drop action from being indiscriminate:
environment = 'dev'at10,environment = 'staging'at5, andenvironment = 'production'at1. Higher values drop first, so production logs now drop last. She setdefault_priority: 5so unmatched traffic wouldn’t inherit the default of10and vanish first. - An
ALERT_WARNthreshold onTRACE_PERSISTED_BYTESwith aROLLING_1_DAY_VOLUMEwindow covered the traces. Trace resource groups reject theDROPaction, so visibility was the only threshold she could set there.
Chronoctl example
Shaping instead of dropping
The thresholds were a backstop. They drop data only after consumption has already run past a limit, and priorities control the order of those drops rather than the value of the data inside each priority. To bring consumption back under the license limit, Nadia needed shaping rules. She returned to the pattern on the Logs Usage page and clicked Create control rule for this pattern, which opened the control rule editor with the pattern’s filter already in place. The editor previewed the logs that filter matched, so she could confirm it caught the retry pattern and nothing adjacent. The obvious rule was to drop the retry logs. Before writing it, she messaged the engineer who had queried the pattern, and learned that the retry count itself was worth keeping, even though nobody wanted the log lines. Nadia chose Emit metric rather than Drop logs. The rule converted each matching log into an increment on a counter namedcheckout_payment_retry_total,
labeled with service and environment, and she selected
Drop original log after emitting metric. A 2.8 TB per day log pattern became a
metric counter, and the signal the engineer cared about survived the reduction.
She confirmed the replacement in
Metrics Explorer:
checkout-service and applied a
behavior that lowered the sampling rate
for non-production spans, previewing the behavior against the dataset before assigning
it.
Proof the changes held
By 2:30 PM, Nadia wanted evidence rather than intent. She looked for three things, and got the last of them the following morning. On the Consumption page, the Per second window for the team partition had fallen from 46 MB per second to 13 MB per second, a daily rate of 1.1 TB rather than 4 TB. The current daily window had already spent its budget by then, so the Budgets panel didn’t clear its status badge until the window reset at midnight UTC. The 2 TB drop threshold had tripped a few minutes past noon, as she expected at 9:42 AM, but the priorities she added in the meantime decided what went. She set Group by to Consumed vs dropped to see the shape of the drops. The only dropped data was in priority10 and priority 5, in the staging and development partitions,
exactly where her priorities said it should be. Production had dropped nothing.
The next morning, the retry pattern on the Logs Usage page reported 24 GB
instead of 2.8 TB. That page is where a log control rule’s impact shows up, as
described in Review shaping rules impact. The remainder
came from retry messages whose text varied enough to fall outside the rule’s filter.
Back on the generated monitor, she left a comment for the
Commerce Platform team:
“Your daily log budget breach came fromThen she shared the notebook, which held the two Logs Explorer panels along with her notes on the log line, the deploy, the pattern volume before and after, and the new threshold values.LOG_LEVEL=debugshipping in checkout-service v3.42.0, left over from the v3.41.0 postmortem. The retry logs are now an emitted metric,checkout_payment_retry_total. Staging and development have their own partitions and instant-rate budgets, so a verbose logger outside production won’t spend the team’s production allowance again. Turn the log level back down in the next release anyway.”