Create metric pools

Create metric pools

Create metric pools with Terraform. Chronosphere uses dry run mode by default. Any applied quota configuration appears in the Metrics Quotas Dashboard and shows you how the pool's traffic interacts with the pool's quota, without penalizing that pool if the system goes over its limit. Use dry run mode to experiment with different pool configurations and quota levels before you enable enforcing mode.

To create metrics quotas, create a set of pools and for each pool provide a:

  • name: Name of the pool.
  • allocation: Allocated quota, as a percentage of your persisted writes license (percent_of_license). The API supports values greater than or equal to 0.001%.
  • match_rules: Match criteria based on a single label/value pair.

If you need to combine Prometheus and Graphite metrics into a single pool, you can specify multiple match criteria, but should select a single Prometheus label and a single Graphite label. If multiple criteria are specified only one needs to match.

In addition to any named metric pools you define, all instances have a Default pool. You can set the quota for the default pool, but not the name or match criteria. The Default pool matches any metric that isn't matched by another pool.

Pools' quota allocations must add up to 100% to take full advantage of your persisted writes license.

Set the Default pool quota to at least 1% to preserve the visibility of metrics that don't fit into a current pool quota.

Pool configurations use the chronosphere_resource_pools_config type followed by a name in a resource declaration.

For example, the following definition sets up two new pools alongside the default pool. It sets the default pool quota to 10%, and creates two other pools that use 40% and 50% of the remaining quota, respectively.

resource "chronosphere_resource_pools_config" "default" {
  default_pool {
    allocation {
      percent_of_license = 10
    }
  }

  pool {
    name = "Team1"
    allocation {
      percent_of_license = 40
    }

    # Include metrics into this pool if either rule matches.
    # In this example, the first rule matches Prometheus metrics and the
    # second rule matches Graphite metrics.
    match_rules = [
      "UsageTagA:{value1,value2,value3}",
      "__g2__:value4"
    ]
  }

  pool {
    name = "Team2"
    allocation {
      percent_of_license = 50
    }
    match_rules = ["UsageTagA:{value5}"]
  }
}

Some fields were changed and deprecated. Changes include:

  • pools renamed to pool
  • match_rule renamed to match_rules

Older field names remain supported during a transition period.

Add the definition to a Terraform file and use terraform apply to create the resource.

Best practices

To keep penalty behavior and cost accounting transparent and predictable, pools should be hard partitions of your system, with no one time series matching more than one pool. The following processes help ensure pools have the correct data:

  • Chronosphere recommends selecting a single usage tag as the pool assignment mechanism. Picking a single tag reduces the possibility where one pool matches serviceX, and a second pool matches environmentY, where time series might match either or both definitions.
  • Use exact match values for the selected label to decrease the chances of other tags or a regex match allowing a time series to fit into more than one pool.
  • Chronosphere uses match-ordering in pools. If a time series matches more than one pool, it becomes part of the first pool in the list that it matches. This doesn't change that a time series might match more than one pool's criteria, but it does ensure that a time series is accounted for consistently in a single pool.

Priorities

This feature isn't available to all Chronosphere users and might not be visible in your app. For information about enabling this feature in your environment, contact your salesperson or technical account manager.

If you have set up metrics quotas and your system goes over its persisted writes license, Chronosphere drops metrics from pools that exceed their respective quotas until all pools meet their quotas. Chronosphere penalizes only pools that exceed their persisted writes quota. To more selectively decide which metrics within a given pool to drop first during a penalty scenario, specify priorities for each pool.

Assign priorities to metrics within a pool using Terraform, similar to pool creation in Terraform. Users can assign priorities based on a single additional usage tag/value pair.

For example, if the usage tag service was used to assign metrics into a pool, you might choose the usage tag environment to assign priorities to metrics within that pool.

  • High: Metrics dropped last.
  • Low: Metrics dropped first.
  • Default: Metrics dropped after Low priority metrics, but before High priority metrics.

When defining priorities, you may specify either high_priority_match_rules, low_priority_match_rules, or both. Metrics which aren't captured in the high or low priority groups will be assigned the default priority.

If all metrics in a pool are associated with a single priority (for instance, all metrics are assigned to the high priority group), the penalty behavior for that pool when the system is over the persisted writes quota, is the same as if all metrics in that pool have the default priority. Priorities assess against their own pool, and not across different pools.

Pool priorities support glob syntax.

The following example is a Terraform definition to implement metrics pool priorities:

  pool {
    name = "Team1"
    allocation {
      percent_of_license = 40
    }
 
    match_rules = [
      "UsageTagA:{value1,value2,value3}",
      "__g2__:value4"
    ]
 
    priorities {
      high_priority_match_rules = ["UsageTagB:{value1}"]
      low_priority_match_rules = ["UsageTagB:{value2}"]
    }
  }