Manage metric pools

In the Metrics Quotas page of Chronosphere Observability Platform, you can review current and preview potential impacts to your quota allocations by pool. Use this feature to shape and plan your pool sizes and future usage.

To get started with pools, learn about how to define a pool.

Add a metric pool

Select from the following methods to add a metric pools. You can define a pool in Observability Platform, but must use Terraform to apply the changes.

Terraform pool example

The following code is an example of a Terraform file used to create quotas and priorities.

This code is an example and should not be used directly. This example uses the following variables in place of a specific name:

• MY_RESOURCE is a resource in your system.
• MY_SERVICE is the name of a service in your system.

resource "MY_RESOURCE_resource_pools_config" "resource_pools" {
  default_pool {
 
    priorities {
      high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"]
      low_priority_match_rules  = ["MY_RESOURCE_k8s_cluster:rc*"]
    }
  }
 
  pool {
    name = "Tracing Services"
 
    allocation {
      percent_of_license = 10
 
      fixed_value {
        license = "PERSISTED_WRITES_STANDARD"
        value = 6500
      }
 
      fixed_value {
        license = "PERSISTED_WRITES_HISTOGRAM"
        value = 2500
      }
    }
 
    match_rules = ["service:{spanhandler,traceingester}"]
 
    priorities {
      high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"]
      low_priority_match_rules  = ["MY_RESOURCE_k8s_cluster:rc*"]
    }
  }
 
  pool {
    name = "MY_SERVICE Services"
 
    allocation {
      percent_of_license = 25
    }
 
    match_rules = ["service:MY_SERVICE*"]
 
    priorities {
      high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"]
      low_priority_match_rules  = ["MY_RESOURCE_k8s_cluster:rc*"]
    }
  }
 
  pool {
    name = "Gateway Services"
 
    allocation {
      percent_of_license = 4
    }
 
    match_rules = ["service:gateway*"]
 
    priorities {
      high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"]
      low_priority_match_rules  = ["MY_RESOURCE_k8s_cluster:rc*"]
    }
  }
}

Edit a pool

Select from the following methods to edit pools. You can also configure global priorities to change global pool quota configurations by metric label.

Understanding pool usage

The Pools section of the Configure Quotas page describes what pools you have, and how they’re configured. This includes:

• Pool name: The display name of the pool.
• Data matching: The label values the pool matches.
• Allocation: The percentage or Data Points Per Second (DPPS) of total traffic guaranteed to the pool before it might be penalized.
• Consumption: The percentage or DPPS of total traffic the pool is consuming over the selected time range.

Quota allocations and consumption

The Quota Allocations vs Current Consumption graph is a bar chart used to visualize the current quota allocation in Data Points Per Second (DPPS) for each pool, and what the pool is actually consuming. The top bar for each pool is the Allocation. The second bar is the Current Consumption for the pools. Using these two bars, you can determine consumption in relation to the allocated quota. Point to any group of bars to display a dialog with exact values.

Quota consumption by pools (per second)

The Quota Consumption by pools graph separates the pools so you can see each pool’s Average consumption and its Current quota limit.

Understand quota consumption trends

The Quota Consumption graph is a running line of data points and quota limits, where the previous graph displays only a single value. Point to any point on the graph to see exact data. Drag in a graph to focus in on the selected time period.

Preview quota allocations

The Pools include a group of text fields corresponding to each created pool. These boxes contain values with the assigned percentage (%) or DPPS for each pool. Use these to set your general pool allocations.

To preview a new quota allocation, change a number in the box for the pool to be updated. Click outside the boxes to update the total.

Quota settings must meet the following criteria:

• Quotas must add up to 100%. Changing any pool’s quota causes a related change in the default pool to ensure total quota is 100%.
• The UI supports values greater than or equal to 0.01%.
• The API supports values greater than or equal to 0.001%.

Changing an assigned quota displays a third bar in the Quota consumption by pools chart. Use the new bar to determine if new quota assignments meet the needs of each of your pools.

If one pool is consistently over quota and the other pools aren’t, use the preview to adjust assigned quotas to better meet the needs of each pool.

Click the Reset quotas icon to return to the existing configuration.

Delete a pool

Select from the following methods to delete pools.

Best practices

To keep penalty behavior and cost accounting transparent and predictable, pools should be hard partitions of your system, with no one time series matching more than one pool. The following processes help ensure pools have the correct data:

• Chronosphere recommends selecting a single usage tag as the pool assignment mechanism. Picking a single tag reduces the possibility where one pool matches serviceX, and a second pool matches environmentY, where time series might match either or both definitions.
• Use exact match values for the selected label to decrease the chances of other tags or a regular expression match allowing a time series to fit into more than one pool.
• Chronosphere Observability Platform uses match-ordering in pools. If a time series matches more than one pool, it becomes part of the first pool in the list that it matches. A time series might match more than one pool’s criteria, but a first-match policy ensures that a time series is accounted for consistently in a single pool.
• If you see a pool that doesn’t match the expected penalty behavior, open the pool in the profiler and compare it with the Terraform configuration file. A match rule value might be incorrect.

Create an alert for a pool in penalty

When a pool is in a penalty state, it might drop metrics to reduce usage. For higher priority pools, this can result in the loss of important data. To reduce or prevent data loss, create a monitor to alert on pool usage and notify the appropriate team to take preventative action.

1. Create a notification policy with criteria like warn if value is > 90% for 5 minutes.

2. Create a notifier and align with your internal alerting policies to route the alerts to the right team.

3. Create a monitor.

4. In the monitor query, add the query to return each pool’s percent utilization. You can set up alerts using the chrono_poolstats_count metric. A query that returns each pool’s percent utilization of its quota might look like this:

   (sum by (metrics_class) (rate(chrono_poolstats_count{type="persisted", dropped="no"}[2m]))) /
   ((sum by(metrics_class) (coordinator_scheduler_metrics_class_weight{})) * on() group_left() limit_service_licensed_persist_limit{})

5. In the Signals section, select Per time series (many alerts) as the signal.

6. Define any additional fields, such as a condition and sustain period.

7. Click Save to save the monitor.