Manage metric pools

On the Metrics Quotas page in Chronosphere Observability Platform, review current quota allocations and preview potential changes by pool. Use this feature to plan pool sizes and future usage. To get started with pools, learn about how to define a pool.

Add a metric pool

Select from the following methods to add a metric pool. Changes made in the web interface are previews that you apply using the generated configuration.

You can define up to 20 custom metric pools.

Web
Chronoctl
Terraform
API

Although actual management of pools is handled using Terraform, this interface helps you understand what changes to make to reduce guesswork and repeated updates to your system.To create quota pools, you must have administrative privileges:

In the navigation menu, click Go to Admin and then select Control > Metrics Quotas.
Click Configure Quotas.
Click Add Pool. If Add Pool is unavailable, expand Global Settings, click Edit Global Settings, and select a Quota Configuration Label.
The following fields appear in the drawer. Update editable fields to modify your pool configuration:
- Pool name: (Editable) Change the pool name.
- Data matching: The values the selected pool uses to match data.
  - Quota configuration label: The label matching this pool.
  - Data matching values: (Editable) The specific values for the label, which match this pool. Add a value and press Enter or Space to add it to the list. The value supports glob syntax.
- Observed label consumption: Displays estimated Standard Metrics consumption for the matching label values, broken down by label value. The estimate isn’t restricted to data currently assigned to the pool.
- Quota allocation: (Editable) Set a quota percentage for the selected pool. The drawer displays the equivalent persisted writes Standard Metrics value in data points per second (DPPS). To enter a DPPS value, use the DPPS mode in the Pools table.
- Prioritization: (Conditionally editable) Add the Priority Label and high or low priority values.
  Pools using a global priority setting can’t change their priorities for an individual pool.
In the Quota Allocation chart, select or clear pool names in the legend to include or remove their segments from the Allocation and Current Consumption stacked bars.
Click Done after completing your changes.
Click the Code config tab.
Click Copy to copy the file, or Download to download the file to your computer.
Review the generated configuration. Percentage changes apply to licenses that don’t have fixed-value overrides. Changes entered in DPPS mode update only the PERSISTED_WRITES_STANDARD fixed value. The generated configuration preserves existing fixed values and thresholds for other licenses.
Add the definition to a Terraform file, or create a new Terraform file.
Run this command to apply the resource:
terraform apply

To create a pool using Chronoctl:

Run the following command to generate a sample pool configuration you can use as a template:
chronoctl resource-pools scaffold
In the template, kind: ResourcePools defines the complete pool configuration. The ResourcePools resource is a singleton that contains the default pool and all named pools. If a ResourcePools resource already exists, follow the Chronoctl edit procedure instead of creating another resource.
With a completed definition, submit it with:
chronoctl resource-pools create -f FILE_NAME.yaml
Replace FILE_NAME with the name of the YAML definition file you want to use.

When you run terraform plan to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the Terraform provider documentation.

To create a pool with Terraform:

Create or edit a Terraform file and add the definition by using the chronosphere_resource_pools_config type, followed by a name in a resource declaration.
Run this command to apply the changes:
terraform apply

See the Terraform pool example for a completed pool resource.

To complete this action with the Chronosphere API, use the CreateResourcePools endpoint.The endpoint creates the complete ResourcePools singleton. If the resource already exists, use the UpdateResourcePools endpoint to add a named pool.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Chronoctl pool example

The following code is an example of a Chronoctl resource used to create quotas and priorities. The example creates two pools: one for team_a and one for team_b. Each pool defines license values for persisted writes and matched writes, along with thresholds to manage persisted cardinality.

Chronoctl example

api_version: v1/config
kind: ResourcePools
spec:
  pools:
    - name: team_a
      allocation:
        fixed_values:
          - license: PERSISTED_WRITES_STANDARD
            value: "70000"
          - license: MATCHED_WRITES_STANDARD
            value: "30000"
        priority_thresholds:
          - license: PERSISTED_CARDINALITY_STANDARD
            all_priorities:
              fixed_value: "6000000"
            low_priority:
              fixed_value: "2000000"
      filters:
        - name: team
          value_glob: 'a'
      priorities:
        high_priority_filters:
          - name: env
            value_glob: '*prod*'
        low_priority_filters:
          - name: env
            value_glob: '*dev*'
    - name: team_b
      allocation:
        fixed_values:
          - license: PERSISTED_WRITES_STANDARD
            value: "16000"
          - license: MATCHED_WRITES_STANDARD
            value: "15000"
        priority_thresholds:
          - license: PERSISTED_CARDINALITY_STANDARD
            all_priorities:
              fixed_value: "1750000"
            low_priority:
              fixed_value: "750000"
      filters:
        - name: team
          value_glob: 'b'
      priorities:
        high_priority_filters:
          - name: env
            value_glob: '*prod*'
        low_priority_filters:
          - name: env
            value_glob: '*dev*'

Terraform pool example

The following code is an example of a Terraform file used to create pools and priorities. This code is an example and shouldn’t be used directly. This example uses the following variables in place of a specific name:

MY_RESOURCE is a resource in your system.
MY_SERVICE is the name of a service in your system.

Terraform example

resource "MY_RESOURCE_resource_pools_config" "resource_pools" {
  default_pool {

    priorities {
      high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"]
      low_priority_match_rules  = ["MY_RESOURCE_k8s_cluster:rc*"]
    }
  }

  pool {
    name = "Tracing Services"

    allocation {
      # Applies to each license that doesn't have a fixed-value override.
      percent_of_license = 10
    }

    match_rules = ["service:{spanhandler,traceingester}"]

    priorities {
      # Optional. Match rules are filters that define which metrics are high or
      # low priority. Any metric that matches at least one filter is considered
      # high or low priority, depending on the defined priority. High priority
      # match rules take precedence over low priority ones. If a metric
      # matches both a high and low priority rule, it's considered a high
      # priority metric. When the license limit is exceeded, high priority
      # metrics are dropped last, and low priority metrics are dropped first.
      # This behavior applies to only persisted writes and matched writes, but
      # doesn't apply to persisted cardinality.
      high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"]
      low_priority_match_rules  = ["MY_RESOURCE_k8s_cluster:rc*"]
    }
  }

  pool {
    name = "Collector service"
    allocation {
      percent_of_license = 8
      # Optional. For supported licenses, defines thresholds with strict limits
      # for when to drop new consumption of the license for a pool. Currently, only
      # `PERSISTED_CARDINALITY_STANDARD` and `PERSISTED_CARDINALITY_HISTOGRAM` are
      # supported.
      priority_thresholds {
        license = "PERSISTED_CARDINALITY_STANDARD"
        # Threshold limit that defines when to drop new metrics in the pool. This
        # threshold applies to all priorities of metrics: high, default, and low.
        # This field must be set, and have a value equal to or greater than other
        # priority fields.
        all_priorities {
          percent_of_pool_allocation = 100
        }
      }
      priority_thresholds {
        license = "PERSISTED_CARDINALITY_HISTOGRAM"
        all_priorities {
          percent_of_pool_allocation = 100
        }
      }
    }
    match_rules = ["service:{${join(",", local.collector_services)}}"]
    priorities {
      high_priority_match_rules = ["chronosphere_k8s_cluster:prod*"]
      low_priority_match_rules  = ["chronosphere_k8s_cluster:rc*"]
    }
  }

  pool {
    name = "MY_SERVICE Services"

    allocation {
      percent_of_license = 25
    }

    match_rules = ["service:MY_SERVICE*"]

    priorities {
      high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"]
      low_priority_match_rules  = ["MY_RESOURCE_k8s_cluster:rc*"]
    }
  }

  pool {
    name = "Gateway Services"

    allocation {
      percent_of_license = 4
    }

    match_rules = ["service:gateway*"]

    priorities {
      high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"]
      low_priority_match_rules  = ["MY_RESOURCE_k8s_cluster:rc*"]
    }
  }
}

Edit a pool

Select from the following methods to edit pools. You can also configure global priorities to change global pool quota configurations by metric label.

The visual editor requires one quota configuration label across all pools and one prioritization label within each pool. Edit configurations that use multiple matching or prioritization labels directly in Terraform.

Web
Chronoctl
Terraform
API

To edit an existing pool:

In the navigation menu, click Go to Admin and then select Control > Metrics Quotas.
Click Configure Quotas.
Click any row in the Pools table to open the Edit Pool drawer. The Edit Pool drawer contains information specific to the selected pool. These fields match the Add Pool screen, and some values can be edited.
Make any necessary changes, and then click Done after completing your changes.
Click the Code config tab.
Click Copy to copy the file, or Download to download the file to your computer.
Review the generated configuration to confirm that existing fixed values and priority thresholds are preserved.
Add the definition to a Terraform file, or create a new Terraform file.
Run this command to apply the resource:
terraform apply

To edit a pool using Chronoctl:

View your existing metrics pools YAML definition with the chronoctl resource-pools read command:
chronoctl resource-pools read
Copy the YAML definition and save it to a new file ending in .yaml.
Modify the YAML definition properties and apply the changes:
chronoctl resource-pools update -f FILE_NAME.yaml
Replace FILE_NAME with the name of the YAML definition file you want to use.

To edit a pool using Terraform:

Create or edit a Terraform file that updates the resource’s existing properties.
Run this command to apply the changes:
terraform apply

You can also use the Code Config tool to view and edit the resource pool Terraform representation:

Click the Code Config tab.
Make changes to the resource pool definition.
Click Copy to copy the file, or Download to download the file to your computer.
Add the definition to a Terraform file, or create a new Terraform file.
Run this command to apply the resource:
terraform apply

To complete this action with the Chronosphere API, use the UpdateResourcePools endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Understanding pool usage

The Pools section of the Configure Quotas page describes what pools you have, and how they’re configured. This includes:

Pool name: The display name of the pool.
Data matching: The label values the pool matches.
Allocation: The percentage or DPPS of total traffic guaranteed to the pool before it might be penalized.
Consumption: The percentage or DPPS of total traffic the pool is consuming over the selected time range.

Allocation and consumption DPPS values describe Standard Metrics License allocation and consumption. Histogram Metrics License allocation and consumption aren’t included in the pool’s reported DPPS.

Quota allocations and consumption

The Quota Allocations vs Current Consumption graph compares two stacked bars. The Allocation bar segments the current quota allocation by pool. The Current Consumption bar segments current consumption by pool. Select pool names in the legend to filter both bars.

Quota consumption by pools (per second)

The Quota Consumption by pools graph compares each pool’s displayed consumption with its current quota limit. Changing an allocation adds the New Quota Limit series.

Understand quota consumption trends

The Quota Consumption graph plots each pool’s consumption over time. Point to a data point to display its value. Drag across the graph to focus on a time period.

Preview quota allocations

The Pools include a group of text fields corresponding to each created pool. These boxes contain values with the assigned percentage (%) or DPPS for each pool. Use these to set your general pool allocations. To preview a new quota allocation, change a number in the box for the pool to be updated. Click outside the boxes to update the total. Quota settings must meet the following criteria:

In the web interface, changing a named pool’s quota adjusts the default pool so that all allocations total 100%.
The Add Pool and Edit Pool drawers support values greater than or equal to 0.01%. Allocation fields in the Pools table support positive values with up to three decimal places.
The API supports values from 0% through 100% with up to three decimal places. If the default pool allocation is omitted, named pool allocations can total less than 100%, and the default pool receives the remainder.

Changing an assigned quota displays a third bar in the Quota consumption by pools chart. Use the new bar to determine if new quota assignments meet the needs of each of your pools. If one pool is consistently over quota and the other pools aren’t, use the preview to adjust assigned quotas to better meet the needs of each pool. Click the Reset quotas icon to restore existing allocation values. This action doesn’t undo changes to names, data matching, priorities, added pools, or removed pools.

Delete a pool

Select from the following methods to delete pools.

Web
Chronoctl
Terraform
API

To delete an existing pool:

In the navigation menu, click Go to Admin and then select Control > Metrics Quotas.
Click Configure Quotas.
Click a non-default pool’s row in the Pools table to open the Edit Pool drawer.
At the bottom of the Edit Pool drawer, click Remove Pool. The pool is removed from the preview and the resource definition in the Code config tab.
Click the Code config tab.
Click Copy to copy the file, or Download to download the file to your computer.
Update your Terraform file with the generated definition.
Run this command to apply the change:
terraform apply

To delete a pool using Chronoctl:

View your existing metrics pools YAML definition with the chronoctl resource-pools read command:
chronoctl resource-pools read
Copy the YAML definition and save it to a new file ending in .yaml.
Remove the pool you want to delete from the YAML definition and save the updated file.
Apply the changes:
chronoctl resource-pools update -f FILE_NAME.yaml
Replace FILE_NAME with the name of the YAML definition file you want to use.

To delete the entire ResourcePools definition, use the chronoctl resource-pools delete command:

chronoctl resource-pools delete

This command deletes the entire ResourcePools definition.

To delete a resource that’s managed by Terraform:

Edit your Terraform configuration file to remove the pre-existing resource definition.
Run this command to remove the resource from Observability Platform:
terraform apply

You can also use the Code Config tool to view and edit the resource pool Terraform representation:

Click the Code Config tab.
Make changes to the resource pool definition.
Click Copy to copy the file, or Download to download the file to your computer.
Add the definition to a Terraform file, or create a new Terraform file.
Run this command to apply the resource:
terraform apply

To complete this action with the Chronosphere API, use the DeleteResourcePools endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

This action deletes the entire ResourcePools definition.

Best practices

To keep penalty behavior and cost accounting transparent and predictable, pools should be hard partitions of your system, with no one time series matching more than one pool. The following processes help ensure pools have the correct data:

Chronosphere recommends selecting a single usage tag as the pool assignment mechanism. Picking a single tag reduces the possibility where one pool matches serviceX, and a second pool matches environmentY, where time series might match either or both definitions.
Use exact match values for the selected label. The API rejects an exact label value that’s already assigned to another pool.
Observability Platform uses match ordering for valid glob patterns that overlap. If a time series matches more than one pool, it becomes part of the first matching pool in the list.
If you see a pool that doesn’t match the expected penalty behavior, open the pool in the profiler and compare it with the Terraform configuration file. A match rule value might be incorrect.

Create an alert for a pool in penalty

When a pool is in a penalty state, it might drop metrics to reduce usage. For higher priority pools, this can result in the loss of important data. To reduce or prevent data loss, create a monitor to alert on pool usage and send notifications to the appropriate team.

Create a notifier and align with your internal alerting policies to route the alerts to the right team.
Create a notification policy that connects the notifier to a monitor.
Create a monitor.

In the monitor query, add the query to return each pool’s percent utilization. For example, the following query calculates persisted writes standard metrics utilization and excludes pools with zero allocation:

(
  100 *
  sum by (pool_name) (
    chrono_metrics_persisted_writes_license_dpps_consumed{datapoint_type="standard"}
  )
  /
  sum by (pool_name) (
    chrono_metrics_persisted_writes_license_dpps_capacity{datapoint_type="standard"}
  )
)
and on (pool_name)
(
  sum by (pool_name) (
    chrono_metrics_persisted_writes_license_dpps_capacity{datapoint_type="standard"}
  ) > 0
)

In the Signals section, select Per time series (many alerts) as the signal.
Define the warning condition as greater than 90 and set the sustain period to 5 minutes.
Click Save to save the monitor.

Observability Platform

Telemetry Pipeline

More information

Add a metric pool

Chronoctl pool example

Terraform pool example

Edit a pool

Understanding pool usage

Quota allocations and consumption

Quota consumption by pools (per second)

Understand quota consumption trends

Preview quota allocations

Delete a pool

Best practices

Create an alert for a pool in penalty

​Add a metric pool

​Chronoctl pool example

​Terraform pool example

​Edit a pool

​Understanding pool usage

​Quota allocations and consumption

​Quota consumption by pools (per second)

​Understand quota consumption trends

​Preview quota allocations

​Delete a pool

​Best practices

​Create an alert for a pool in penalty

Add a metric pool

Chronoctl pool example

Terraform pool example

Edit a pool

Understanding pool usage

Quota allocations and consumption

Quota consumption by pools (per second)

Understand quota consumption trends

Preview quota allocations

Delete a pool

Best practices

Create an alert for a pool in penalty