> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Telemetry data budgets
After [defining partitions](/control/consumption/partitions),
[analyzing consumption](/control/consumption/analyze), and making any necessary
adjustments, you can create *budgets*, which are shaping policies that you can attach
to a partition.
Budgets provide guardrails to combat runaway usage and overspending, while providing
the flexibility to enforce accountability at the right level of ownership. Budgets
include *thresholds* that define which actions to take when a threshold is exceeded.
You can also set *priorities* to control the order in which data is dropped when a
drop action is applied, with lower priority values dropped last. For example, a
priority of `10` is dropped first, and a priority of `1` is dropped last.
Priorities are evaluated in match order, and the first priority to match is applied.
All other priorities are ignored. Use the `default_priority` field to set the
priority on requests that don't match any defined priorities.
## Threshold types
Budgets can include *thresholds*, which help manage both anomalous spikes in data and
slow data growth over time, depending on the type. Use different threshold types
depending on the use case.
For each threshold, you can specify an action, such as sending an alert or dropping
data. When you define an alert action on a threshold, such as sending a warning
alert, Observability Platform generates an associated monitor in the
**Chronosphere Managed** [collection](/administer/collections).
Although configuring thresholds is optional, Chronosphere strongly recommends setting
thresholds to protect budgets and avoid overspending.
### Anomalous spikes in real time
To detect data spikes in real time and react accordingly, configure either an
instant-rate threshold or a rolling threshold. These thresholds implement shorter
time windows that are more responsive, but can result in more noise.
Instant-rate thresholds limit the rate of incoming data in bytes per second,
specified as fixed values. With an instant rate threshold, the configured action
(such as sending an alert) triggers when consumption exceeds the configured rate
threshold. For example, if a threshold is exceeded, Observability Platform can send
an alert to notify the team who owns the budget for the offending partition.
Alternatively, you can start dropping data when a threshold value is met so that the
budget isn't exceeded, which would otherwise impact your persisted data limit.
For alert actions on instant-rate thresholds, configure `instant_rate_sustain_secs`
in the `alert_action_config` to control how long consumption must remain above the
threshold before an alert triggers. The default is `0`, which triggers an alert
immediately on any breach.
Rolling thresholds help to detect and manage anomalous spikes in data over a short
period, such as one to three hours. The available rolling threshold types are:
* `ROLLING_30_MINUTE_VOLUME`: Rolling thirty-minute window.
* `ROLLING_1_HOUR_VOLUME`: Rolling one-hour window.
* `ROLLING_3_HOUR_VOLUME`: Rolling three-hour window.
* `ROLLING_1_DAY_VOLUME`: Rolling one-day window.
* `ROLLING_7_DAY_VOLUME`: Rolling seven-day window.
### Slow growth over time
To help detect and manage gradual data growth over a longer period (which can be
daily, weekly, or monthly), use fixed volume-based thresholds. These thresholds
implement longer time windows that are less responsive, but are also less noisy than
instant-rate or rolling thresholds. The available fixed volume threshold types are:
* `HOURLY_VOLUME`: Resets every hour at `HH:00:00` UTC.
* `DAILY_VOLUME`: Resets daily at `00:00:00` UTC.
* `WEEKLY_VOLUME`: Resets weekly on Monday at `00:00:00` UTC.
* `MONTHLY_VOLUME`: Resets on the first of each month at `00:00:00` UTC.
Fixed volume-based thresholds set a limit over a fixed, non-overlapping period, such
as one day. The configured action (such as dropping data) triggers when consumption
over the specified time period exceeds the configured threshold. For example,
dropping data on a daily basis if a budget for a service exceeds a fixed value
threshold of a configured number of bytes.
### Drop data with thresholds
When certain thresholds with a drop action are exceeded, Observability Platform drops
data according to the length of the time window and the threshold type:
* Instant-rate thresholds: Observability Platform accepts requests at a rate of up to
the threshold value, and drops data when the rate is over the threshold in priority
order. Data resumes when the incoming rate falls below the threshold.
* Fixed volume-based thresholds: Observability Platform drops data until the end of
the time window. For example, a daily volume-based threshold that's configured to
drop data at 12:00 AM drops data until 11:59 PM.
* Rolling thresholds: Observability Platform drops data until the data in the
affected budget naturally ages out of the rolling window, and your tenant
consumption is under the configured threshold.
Avoid setting multiple volume-based thresholds with a drop action on the same budget.
For example, setting a daily, weekly, and monthly volume-based threshold provides
considerable overlap, and could result in unwanted noise (such as excessive alerts).
Each combination of action and threshold type must be unique within a budget. For
example, a budget cannot contain two `DROP` thresholds that are both `INSTANT_RATE`.
## Budgeting order
If you attach multiple budgets to every level of your partition hierarchy, a single
request must pass through multiple thresholds before being accepted. Budgets are
evaluated in order, from the most-specific partition that you created, to the
least-specific partition (the global partition). If a budget drops a request, then
data isn't counted in the received rates of any parent budgets.
For example, consider the following partitions and budgets:
* A partition for `global/team-a` has an `instant_rate` threshold of 2 MB per
second.
* A partition for `global/team-a/service-1` has an instant rate threshold of
1 MB per second.
```mermaid actions={false} theme={null}
flowchart TD
global --> team-a
team-a --> service-1
```
If the partition `global/team-a/service-1` receives a huge data spike of 1 GB
per second, then 999 MB of that overage is dropped before it's counted against
the budget for `global/team-a`. Therefore, the `global/team-a` partition receives
only 1 MB per second, and won't be impacted by data drops.
This budget hierarchy ensures that drop data from the `global/team-a/service-1`
partition only, and not from `global/team-a`. The budget that's configured on the
individual service ensures that the team budget isn't impacted. This behavior allows
you to penalize bad actors and not impact the upstream partitions.
## View budgets
Select from one of the following methods to view budgets for partitions.
To use [Chronoctl](/tooling/chronoctl) to return all budgets, use the
`chronoctl consumption-budgets list` command:
```shell theme={null}
chronoctl consumption-budgets list
```
To filter for a specific budget, use the `chronoctl consumption-budgets read`
command:
```shell /SLUG/ theme={null}
chronoctl consumption-budgets read SLUG
```
Replace *`SLUG`* with the slug of the budget you want to display.
To return a list of budgets with the Chronosphere API, use the
[`ListConsumptionBudgets`](/tooling/api-info/definition/operations/ListConsumptionBudgets)
endpoint.
To return a single budget definition, use the
[`ReadConsumptionBudget`](/tooling/api-info/definition/operations/ReadConsumptionBudget) endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
## Create budgets
When creating a budget, complete the following actions:
* Required: Include the slug path of the partition that the budget applies to.
* Optional: Specify an alert notification policy to use for routing alerts.
* Optional (but highly suggested): Define thresholds and actions to take if a
threshold is exceeded.
* Optional: Apply priorities for each of the query conditions that match.
Configure alert actions before enforcing drops. Alerts provide visibility without
impacting persisted data. For each alert you define on a threshold, Observability
Platform generates an associated monitor in the **Chronosphere Managed** collection.
To configure a generated monitor, specify metadata in the `alert_action_config`
section of the budget definition. For example, you can add `annotations` that include
links to runbooks, and specify `labels` that can include metadata like `team` and
`environment`. These labels can be used for routing notifications, such as to
specific teams to notify when an alert triggers. See the
[Chronoctl](#chronoctl-budget-example) and [Terraform](#terraform-budget-example)
examples for details.
You can specify multiple alert actions for a single threshold. For example, define a
warning alert at one value, and then a critical alert at a higher value. However,
Observability Platform generates only one monitor for each
[threshold type](#threshold-types) in a budget. This means that even if you define
multiple alerts for a threshold type, such as an instant rate threshold, only one
monitor is created.
### Prerequisites
Before creating budgets, [define partitions](/control/consumption/partitions).
Use the slug or ID of your partition to assign budgets.
Because budgets use Chronosphere alerts to send notifications when thresholds are
exceeded, you need to define [notification policies](/investigate/alerts/notifications/policies)
to reference in your budgets.
### Create partition budgets
Use one of the following methods to create budgets for partitions.
To use [Chronoctl](/tooling/chronoctl) to create a budget, use the
`chronoctl consumption-budgets create` command:
```shell theme={null}
chronoctl consumption-budgets create
```
1. Run the following command to generate a sample budget configuration you can use as
a template:
```shell theme={null}
chronoctl consumption-budgets scaffold
```
In the template, `kind: ConsumptionBudget` defines an individual budget.
You can redirect the results (using the redirection operator `>`) to a file for
editing.
2. With a completed definition, submit it with:
```shell /FILE_NAME/ theme={null}
chronoctl consumption-budgets create -f FILE_NAME
```
Replace *`FILE_NAME`* with the name of the YAML definition file you want to use.
See the [Chronoctl example](#chronoctl-budget-example) for a completed budget
resource.
When you run `terraform plan` to generate an execution plan, Chronosphere automatically
tests configurations that include notification policies by submitting them as dry runs.
For details, see the
[Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs)
documentation.
To create a budget with [Terraform](/tooling/infrastructure/terraform):
1. Create or edit a Terraform file and add the definition by using the
`chronosphere_consumption_budget` type.
2. Enter a `consumption_config_id` for the budget.
3. Include the `notification_policy_id` that references the ID of the
notification policy you want to use for routing alerts.
4. Include the `partition_slug_path` that references the slug of the partition you
want to assign this budget to.
5. Define a `threshold` condition to send alerts.
6. Define a `threshold` condition to drop data.
7. Run this command to apply the changes:
```shell theme={null}
terraform apply
```
See the [Terraform budget example](#terraform-budget-example) for a completed
budget resource.
To complete this action with the Chronosphere API, use the
[`CreateConsumptionBudget`](/tooling/api-info/definition/operations/CreateConsumptionBudget)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
### Next steps
After creating budgets, use the Consumption page to monitor and
[analyze consumption](/control/consumption/analyze) for the budget you added to your
partition:
* Review relative consumption trends: In the **Consumption** panel group, use the
dropdown menus to view consumption trends across different time periods to
understand what's driving data growth.
* Detect budgets approaching thresholds: Use the **Budgets** panel group to view data
matching the selected partition against either the maximum consumption rate set in
your contract for the global partition, or a custom threshold for custom
partitions.
* Observe drops from rate-limiting thresholds per partition: Use the
**Consumed vs dropped** option in the **Group by** menu to view the rate at which
data was dropped by a threshold for the selected partition, broken down by priority
(if configured).
### Chronoctl budget example
The following Chronoctl example adds a budget for the Ordering Service, which is
included in the
[partition hierarchy](/control/consumption/partitions#partition-hierarchy) diagram.
See the
[Chronoctl example](/control/consumption/partitions#chronoctl-partition-example)
that implements a partition for this service.
This budget is scoped to the parent partition `global/ordering-service`, and defines
the following thresholds:
* An alert warning (`ALERT_WARN`) with a threshold of 4,500 bytes per second.
* A critical warning (`ALERT_CRITICAL`) with a threshold of 5,000 bytes per second.
* A drop action (`DROP`) with a threshold of 5,000 bytes per second.
* A drop action (`DROP`) for daily volume, with a threshold of 90,9997 bytes.
Each `priority` controls which data gets dropped first when the `DROP` action
triggers. Higher priority values are dropped first, and lower priority values are
dropped last:
* The query `"env = 'dev'"` has a priority of `10`, so it's dropped first.
* The query `"env = 'staging'"` has a priority of `5`, so it's dropped next.
* The query `"env = 'prod'"` has a priority of `1`, so it's dropped last.
```yaml theme={null}
api_version: v1/config
kind: ConsumptionBudget
spec:
slug: "ordering-service-logs"
name: "Ordering Service Logs"
partition_slug_path: "global/ordering-service"
alert_action_config:
annotations:
runbook: "https://wiki.example.com/runbooks/log-budget-alerts"
team: "ordering-engineering"
description: "Custom description for log budget alerts"
labels:
severity: "high"
environment: "production"
service: "ordering-service"
instant_rate_sustain_secs: 60
priorities:
- filters:
log_filter:
query: "env = 'dev'"
priority: 10
- filters:
log_filter:
query: "env = 'staging'"
priority: 5
- filters:
log_filter:
query: "env = 'prod'"
priority: 1
thresholds:
- action: ALERT_WARN
instant_rate:
fixed_value_per_sec: 4500
type: INSTANT_RATE
- action: ALERT_CRITICAL
instant_rate:
fixed_value_per_sec: 5000
type: INSTANT_RATE
- action: DROP
instant_rate:
fixed_value_per_sec: 5000
type: INSTANT_RATE
- action: DROP
volume:
fixed_value: 909997
type: DAILY_VOLUME
notification_policy_slug: "budget-alerts-slack"
resource: LOG_PERSISTED_BYTES
```
### Terraform budget example
The following Terraform example adds a budget for the Ordering Service, which is
included in the
[partition hierarchy](/control/consumption/partitions#partition-hierarchy) diagram.
See the
[Terraform example](/control/consumption/partitions#terraform-partition-example)
that implements a partition for this service.
This budget is scoped to the parent partition `global/ordering-service`, and defines
the following thresholds:
* An alert warning (`ALERT_WARN`) with a threshold of 4,500 bytes per second.
* A critical warning (`ALERT_CRITICAL`) with a threshold of 5,000 bytes per second.
* A drop action (`DROP`) with a threshold of 5,000 bytes per second.
* A drop action (`DROP`) for daily volume, with a threshold of 90,9997 bytes.
Each `priority` controls which data gets dropped first when the `DROP` action
triggers. Higher priority values are dropped first, and lower priority values are
dropped last:
* The query `"env = 'dev'"` has a priority of `10`, so it's dropped first.
* The query `"env = 'staging'"` has a priority of `5`, so it's dropped next.
* The query `"env = 'prod'"` has a priority of `1`, so it's dropped last.
The `notification_policy_id` references a notification policy created in Terraform.
To reference a notification policy created in Observability Platform, enclose the
value in double quotes.
```terraform theme={null}
resource "chronosphere_consumption_budget" "ordering_service_budget" {
consumption_config_id = chronosphere_consumption_config.example.id
name = "Ordering Service Logs"
slug = "ordering-service-logs"
notification_policy_id = chronosphere_notification_policy.budget_alerts_slack.id
partition_slug_path = "global/ordering-service"
# Alert action configuration
alert_action_config {
# Additional annotations to set on generated monitors
# These override settings on default annotations if they match default keys
annotations = {
"runbook" = "https://wiki.example.com/runbooks/log-budget-alerts"
"team" = "ordering-engineering"
"description" = "Custom description for log budget alerts"
}
# Additional labels for notification routing
labels = {
"severity" = "high"
"environment" = "production"
"service" = "ordering-service"
}
# Determines how long instant rate consumption must sustain above the threshold
# before triggering an alert
# Default is 0 (any consumption over threshold triggers immediately)
instant_rate_sustain_secs = 60
}
# Threshold configurations
threshold {
# Instant-rate thresholds limit the rate of incoming data in bytes per second,
# specified as fixed values.
action = "DROP"
type = "INSTANT_RATE"
instant_rate {
fixed_value_per_sec = 5000
}
}
threshold {
action = "ALERT_WARN"
type = "INSTANT_RATE"
instant_rate {
fixed_value_per_sec = 4500
}
}
threshold {
action = "ALERT_CRITICAL"
type = "INSTANT_RATE"
instant_rate {
fixed_value_per_sec = 5000
}
}
# Fixed volume-based thresholds set a limit over a fixed, non-overlapping period,
# such as one day. This drop action has a threshold of 90,9997 bytes.
threshold {
action = "DROP"
type = "DAILY_VOLUME"
volume {
fixed_value = 909997
}
}
# Priorities determine the drop order
# The default priority applies to requests that don't match any priority filter.
# If unset, the priority defaults to 10 (dropped first).
default_priority = 5
priority {
priority = 1
filter {
log_filter {
query = "env = 'prod'"
}
}
}
priority {
priority = 5
filter {
log_filter {
query = "env = 'staging'"
}
}
}
priority {
priority = 10
filter {
log_filter {
query = "env = 'dev'"
}
}
}
}
```
## Update budgets
Select from one of the following methods to update budgets for available partitions.
Any changes to the `notification_policy` or `alert_action_config` are automatically
updated in the monitor for the associated threshold.
If you remove an existing threshold from a budget that includes an alert action, the
monitor that's generated automatically from that alert action is deleted from
Observability Platform.
To update budgets with [Chronoctl](/tooling/chronoctl), use the
`consumption-budgets update` command:
```shell theme={null}
chronoctl consumption-budgets update
```
To edit a budget using [Terraform](/tooling/infrastructure/terraform):
1. Create or edit a Terraform file that updates the resource's existing properties.
2. Run this command to apply the changes:
```shell theme={null}
terraform apply
```
To complete this action with the Chronosphere API, use the
[`UpdateConsumptionBudget`](/tooling/api-info/definition/operations/UpdateConsumptionBudget)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
## Delete budgets
Select from one of the following methods to delete budgets for partitions.
Users can modify Terraform-managed resources only by using Terraform.
[Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources).
To delete budgets with [Chronoctl](/tooling/chronoctl), use the
`consumption-budgets delete` command:
```shell theme={null}
chronoctl consumption-budgets delete SLUG
```
Replace *`SLUG`* with the slug of the budget you want to delete.
To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform):
1. Edit your Terraform configuration file to remove the pre-existing resource
definition.
2. Run this command to remove the resource from Observability Platform:
```shell theme={null}
terraform apply
```
To complete this action with the Chronosphere API, use the
[`DeleteConsumptionBudget`](/tooling/api-info/definition/operations/DeleteConsumptionBudget)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.