Log budgets

After defining log partitions, analyzing consumption, and making any necessary adjustments, you can create budgets, which are optional shaping policies that you can attach to a partition.

Budgets provide guardrails to combat runaway usage and overspending, while providing the flexibility to enforce accountability at the right level of ownership. Budgets include thresholds that define which actions to take when a threshold is exceeded.

For example, if a threshold is exceeded, you can send an alert to notify the team who owns the budget for the offending partition. Alternatively, you can start dropping data when a threshold is value is met so that the budget isn’t exceeded, which would otherwise impact your persisted data limit.

You can also set priorities to control the order in which data is dropped when a drop action is applied, with lower priority values dropped last. For example, a priority of 10 is dropped first, and a priority of 1 is dropped last.

Priorities are evaluated in match order, and the first priority to match is applied. All other priorities are ignored. Use the default_priority field to set the priority on requests that don’t match any defined priorities.

Budgeting order

If you attach multiple budgets to every level of your partition hierarchy, a single request must pass through multiple thresholds before being accepted. Budgets are evaluated in order, from the most-specific partition that you created, to the least-specific partition (the global partition). If a budget drops a request, then data isn’t counted in the received rates of any parent budgets.

For example, consider the following partitions and budgets:

• A partition for global/team-a has an instant_rate threshold of 2 MB per second.
• A partition for global/team-a/service-1 has an instant rate threshold of 1 MB per second.

If the partition global/team-a/service-1 receives a huge data spike of 1 GB per second, then 999 MB of that overage is dropped before it’s’counted against the budget for global/team-a. Therefore, the global/team-a partition receives only 1 MB per second, and won’t be impacted by data drops.

This budget hierarchy ensures that drop data from the global/team-a/service-1 partition only, and not from global/team-a. The budget that’s configured on the individual service ensures that the team budget isn’t impacted. This behavior allows you to penalize bad actors and not impact the upstream partitions.

View budgets

Select from one of the following methods to view budgets for log partitions.

chronoctl consumption-budgets list

chronoctl consumption-budgets read SLUG

Create budgets

Log budgets require the following information:

• Include the slug of the partition that the budget applies to.
• Specify an alert notification policy to use for routing alerts.
• Define thresholds and actions to take if a threshold is exceeded.
• Apply priorities for each of the query conditions that match.

Configure alert actions before enforcing drops. Alerts provide visibility without impacting persisted data.

Prerequisites

Before creating budgets, define log partitions. Use the slug or ID of your partition to assign budgets.

Because budgets use Chronosphere alerts to send notifications when thresholds are exceeded, you need to define notification policies to reference in your budgets.

Create partition budgets

Use one of the following methods to create budgets for partitions.

chronoctl consumption-budgets create

chronoctl consumption-budgets scaffold

Next steps

After creating budgets, use the Logging License Consumption dashboard to monitor and analyze consumption for the budget you added to your partition:

• Detect budgets approaching thresholds: Use the Received vs Consumed Per Second panel to view data matching the selected partition against either the maximum consumption rate set in your contract for the global partition, or a custom threshold for custom partitions.
• Observe drops from rate-limiting thresholds per partition: Use the Dropped By Partition Per Second panel to view the rate at which data was dropped by a threshold for the selected partition, broken down by priority if applicable.
• Debug the source of consumption: In the Explore panel group, click the Logs Explorer link to investigate source of consumption by various fields.

Chronoctl budget example

The following Terraform example adds a budget for the Ordering Service, which is included in the partition hierarchy diagram. See the Chronoctl example that implements a partition for this service.

This budget is scoped to the parent partition global/ordering-service, and defines two thresholds:

• An alert warning (ALERT_WARN) with a threshold of 4,500 bytes per second.
• A critical warning (ALERT_CRITICAL) with a threshold of 5,000 bytes per second.
• A drop action (DROP) with a threshold of 5,000 bytes per second.

Each priority controls which data gets dropped first when the DROP action triggers. Higher priority values are dropped first, and lower priority values are dropped last:

• The query "env = 'dev'" has a priority of 10, so it’s dropped first.
• The query "env = 'staging'" has a priority of 5, so it’s dropped next.
• The query "env = 'prod'" has a priority of 1, so it’s dropped last.

api_version: v1/config
kind: ConsumptionBudget
spec:
    slug: "ordering-service-logs"
    name: "Ordering Service Logs"
    partition_slug_path: "global/ordering-service"
    priorities:
         - filters:
              log_filter:
                query: "env = 'dev"
           priority: 10
         - filters:
              log_filter:
                query: "env = 'staging'"
           priority: 5
         - filters:
              log_filter:
                query: "env = 'prod'"
           priority: 1
    thresholds:
        - action: ALERT_WARN
          instant_rate:
            fixed_value_per_sec: 4500
          type: INSTANT_RATE
        - action: ALERT_CRITICAL
          instant_rate:
            fixed_value_per_sec: 5000
          type: INSTANT_RATE
        - action: DROP
          instant_rate:
            fixed_value_per_sec: 5000
          type: INSTANT_RATE
    notification_policy_slug: "budget-alerts-slack"
    resource: LOG_PERSISTED_BYTES

Terraform budget example

The following Terraform example adds a budget for the Ordering Service, which is included in the partition hierarchy diagram. See the Terraform example that implements a partition for this service.

This budget is scoped to the parent partition global/ordering-service, and defines two thresholds:

• An alert warning (ALERT_WARN) with a threshold of 4,500 bytes per second.
• A drop action (DROP) with a threshold of 5,000 bytes per second.

Each priority controls which data gets dropped first when the DROP action triggers. Higher priority values are dropped first, and lower priority values are dropped last:

• The query "env = 'dev'" has a priority of 10, so it’s dropped first.
• The query "env = 'staging'" has a priority of 5, so it’s dropped next.
• The query "env = 'prod'" has a priority of 1, so it’s dropped last.

resource "chronosphere_consumption_budget" "ordering_service_budget" {
  consumption_config_id = chronosphere_consumption_config.example.id
  name                  = "Ordering Service Logs"
  slug                  = "ordering-service-logs"
 
  resource = "LOG_PERSISTED_BYTES"
 
  notification_policy_slug = "budget-alerts-slack"
 
  partition_slug_path = "global/ordering-service"
 
  threshold {
    action = "DROP"
    type = "INSTANT_RATE"
    instant_rate_threshold {
      fixed_value_per_sec = 5000
    }
  }
 
  threshold {
    action = "ALERT_WARN"
    type = "INSTANT_RATE"
    instant_rate_threshold {
      fixed_value_per_sec = 4500
    }
  }
 
  priority {
    priority = 1
    filter {
      condition {
        log_filter {
          query = "env = 'prod'"
        }
      }
    }
  }
 
  priority {
    priority = 5
    filter {
      condition {
        log_filter {
          query = "env = 'staging'"
        }
      }
    }
  }
 
  priority {
    priority = 10
    filter {
      condition {
        log_filter {
          query = "env = 'dev'"
        }
      }
    }
  }
}

Update budgets

Select from one of the following methods to update budgets for available log partitions.

consumption-budgets update

Delete budgets

Select from one of the following methods to delete budgets for log partitions.

consumption-budgets delete