Capacity licensing concepts - Chronosphere Documentation

In a capacity model, quotas determine which data drops first. You can split the total system-persisted writes per second into per-pool quota allocations. In a capacity model, Observability Platform defines two types of metric licenses: the Standard Metrics License and Histogram Metrics License.

Standard Metrics License

The Standard Metrics License measures aggregations, persisted writes, and persisted cardinality license consumption for the following Observability Platform metric types:

Cumulative counter
Delta counter
Gauge

Because Observability Platform aggregates and persists legacy Prometheus histograms and OpenTelemetry explicit bucket layout histograms as cumulative or delta counters, these metrics consume Standard Metrics License capacity.

Histogram Metrics License

The Observability Platform histogram metric type supports both OpenTelemetry exponential histograms and Prometheus native histograms. The Histogram Metrics License measures aggregations, persisted writes, and persisted cardinality license consumption for the following Observability Platform metric types:

Cumulative exponential histogram
Delta exponential histogram

Use the Trends tab of the License Overview page to observe histogram persisted writes, matched writes, and persisted cardinality in the Metrics consumption trends graph.

Capacity license metrics

The following metrics apply in a capacity licensing model.

Matched writes

Matched writes are the number of writes per second being matched for transformation and reshaping by the Observability Platform aggregation tier. A matched write is counted for each data point matched into each aggregator rule, whether rollup or downsampling. If a data point matches one rule, that’s one matched write. If a data point matches two rules, that’s two matched writes. The sum of the matched data points per second per rule equals the total matched writes. Recording rules aren’t considered an aggregation rule for the purpose of counting matched writes. Writes also depend on your Collector scrape interval. Increasing the scrape interval produces fewer writes, but can reduce visibility. See your current Matched Writes level in the License Overview Snapshot in the Metrics Consumption section. On the Trends page, review usage over time in the Metrics Consumption Trends graph.

Matched writes metrics

The following metrics apply to matched writes, which are the number of writes per second being matched for transformation and reshaping by the Observability Platform aggregation tier.

Metric name	Description
`chrono_metrics_matched_writes_license_dpps_limit`	License limit for matched write DPPS by datapoint type.
`chrono_metrics_matched_writes_license_dpps_capacity`	Capacity limit for matched write DPPS by datapoint type.
`chrono_metrics_matched_writes_license_dpps_consumed`	Consumption rate in DPPS of matched write license by datapoint type.

Query the following metric to understand if data is actively being dropped:

chrono_metrics_matched_license_dpps_dropped

Persisted writes

The number of persisted writes to the Observability Platform database consists of the following:

(Number of unaggregated, raw data points written to the database)
+ (Number of aggregated data points written to the database)

If you exceed 100% of your persisted writes capacity limit, data points will be dropped before being persisted. To improve and enhance performance, stability, and features, Observability Platform adds a small number of additional time series to your database. These data points aren’t counted against your license. You can review your current usage in the Persisted writes graph on the License Overview Snapshot page, in the Metrics consumption section. To see changes over time, select Trends, and review the Metrics consumption trends graph.

Persisted writes metrics

The following metrics apply to persisted writes.

Metric name	Description
`chrono_metrics_persisted_writes_license_dpps_limit`	License limit for persisted write DPPS by datapoint type.
`chrono_metrics_persisted_writes_license_dpps_capacity`	Capacity limit for persisted write DPPS by datapoint type.
`chrono_metrics_persisted_writes_license_dpps_consumed`	Consumption rate in DPPS of persisted write license by datapoint type.

For each of these metrics, you can query by datapoint_type, such as histogram or standard. For the _dpps_capacity and _dpps_consumed metrics, you can additionally query by pool_name, and priority. For example, the following query returns the consumption rate in DPPS of your persisted write license for histogram data points on the Auth Services pool. Because priority isn’t specified, the query returns a series for each priority:

chrono_metrics_persisted_writes_license_dpps_consumed{datapoint_type="histogram", pool_name="Auth Services"}

If you wanted to return only high priority histogram data points for the Auth Services pool, specify priority = "high" in your query:

chrono_metrics_persisted_writes_license_dpps_consumed{datapoint_type="histogram", pool_name="Auth Services", priority="high"}

Query the following metric to understand if data is actively being dropped:

chrono_metrics_persisted_license_dpps_dropped{limit="persisted_writes"}

Persisted cardinality

Matched writes and persisted writes measure the rate of data points per second at any given moment in time. Persisted cardinality operates differently, because it’s a cumulative measure that calculates the count of the unique time series of the persisted writes that Observability Platform stores, seen over the last 2.5 hours. This measure is also known as Active Time Series (ATS). Persisted cardinality can be influenced by a change in ingested metrics, or you can use rollup rules to downsample and aggregate metrics before they’re stored. Because this measure is cumulative, reductions in persisted cardinality aren’t reflected immediately, as inactive time series continue counting until they fall outside the 2.5 hour rolling window. Read more about persisted cardinality limits work and how to manage them:

Learn how persisted cardinality limits work
Manage persisted cardinality limits
Avoid persisted cardinality limits

To see persisted cardinality license usage changes over time, in the License overview select Trends, and review the Metrics consumption trends graph.

Persisted cardinality metrics

The following metrics apply to persisted cardinality.

Metric name	Description
`chrono_metrics_persisted_cardinality_license_limit`	License limit for active persisted time series cardinality.
`chrono_metrics_persisted_cardinality_license_capacity`	Capacity limit for active persisted time series cardinality.
`chrono_metrics_persisted_cardinality_license_consumed`	Consumption of the persisted write cardinality limit by datapoint type.

Query the following metric to understand if data is actively being dropped:

chrono_metrics_persisted_license_dpps_dropped{limit="persisted_cardinality"}

How persisted cardinality limits work

Persisted cardinality is comparable to a leaky bucket. Over time, new series can be added until the bucket is full. When the bucket is at maximum capacity, there’s no space for new time series, so they’re rejected. When existing time series expire, they make room for new series. In the following example, the persisted cardinality capacity is five unique time series. The animated image shows the lifecycle of six, unique time series (A, B, C, D, E, and F) as new data points are added, and as other data points expire.

Animated image showing data points being introduced. When the persisted cardinality limit is reached, no more time series are accepted.

As data points are introduced, they’re either accepted or rejected based on whether the persisted cardinality bucket is full (reached maximum capacity), and whether the related time series already exists in the bucket:

If the bucket is at maximum capacity and the series already exists, the data point is accepted.
If the bucket is at maximum capacity and the series doesn’t exist, the series is rejected.

The following table shows how data points A3, E1, and F1 are processed, based on the bucket status:

Data point	Status	Description
A3		Time series A is in the bucket, so data point A3 is accepted.
E1		Time series E isn’t in the bucket, but the bucket has space for one more time series, so data point E1 is accepted in time series E.
F1		Time series F isn’t in the bucket, and the bucket is at capacity, so data point F1 is rejected.

Over time, data points expire based on when they entered the bucket. When data points exceed the 2.5 hour window, they’re excluded from the persisted cardinality bucket. In the example, data points A1 and D1 expired, so they’re excluded from the bucket. When data point C1 expires, it’s also excluded. Because data point C1 is the last data point in time series C, the entire series is removed, making space for a new time series in the bucket.

Data point	Status	Description
A1		Data point A1 expired, so it’s excluded from the bucket.
D1		Data point A1 expired, so it’s excluded from the bucket.
C1		Data point C1 is expiring, so it’s excluded from the bucket.

Manage persisted cardinality limits

If your organization exceeds 100% of their Persisted Cardinality Capacity Limit, data points for any new time series not seen in the last 2.5 hours will be dropped until you’re under this limit. Data points for existing time series will continue to be persisted. Series that are more stable or regularly emitted aren’t at risk of being dropped because they’re always in the system, and aren’t categorized as new series. For example, series that don’t change any labels are considered more stable. To fully resolve a penalty period, the rate of new series must be less than the rate of expiring series. The higher the differential between these rates, the faster the penalty resolves. To manage persisted cardinality limits:

Review the Persisted Cardinality Quotas dashboard, the Usage Dashboard and the Metric Growth dashboard to understand the source of cardinality growth.
Create drop rules and aggregation rules like mapping rules and rollup rules to roll away sources of growth. Old series remain in the cardinality window for 2.5 hrs.
Use the Recommendations page to help identify metrics and labels with no usage or utility over the past 30 days. You can then create drop rules and rollup rules based on the recommendations.

The 2.5 hour expiration window is a rolling window, which means the constant rate of expiring series makes room for an equal rate of new series to be added. This behavior means the penalty period you experience can be much shorter than 2.5 hours. Queries with time ranges that overlap with penalty periods can produce inconsistent or partial results due to the dropping of time series during those periods.

Avoid persisted cardinality limits

Create thresholds on budgets to help manage both anomalous spikes in data and slow data growth over time. Configure actions on each threshold and set a priority to determine what data to drop, and in what order. Use the following tools and techniques to avoid hitting persisted cardinality limits:

Review the Persisted Cardinality Quotas dashboard, the Usage Dashboard and the Metric Growth dashboard to understand the source of cardinality growth.
Learn about different methods to reduce cardinality.
Proactively create drop rules and aggregation rules like mapping rules and rollup rules ahead of potential overages to evict older time series and make room for new ones.
Proactively define thresholds on budgets to better manage persisted cardinality.
If your organization knows which new metrics services are generating, try to control the rate that new series are introduced through smaller, more incremental deploys.

Capacity limits

Capacity limits only apply to capacity pricing. If your organization uses the consumption model, see Manage consumption.

Capacity limits indicate your maximum license capacity for telemetry data in Observability Platform. Exceeding your capacity limits incurs penalties, which can result in dropped data. Dropped data can affect dashboards, alerts, and other reports. Licensing capacity is based on your telemetry types and usage. The license limit indicates your contractual license with Chronosphere. The capacity and license limits display in the Contracts tab of the License Overview page. These limits are broken down into individual limit graphs:

Persisted writes
Matched writes
Persisted cardinality
Histogram persisted writes
Histogram matched writes
Histogram persisted cardinality

Chronosphere recommends creating alerts using the existing capacity limit metrics, which are also used in the License Overview. Use alerts to be notified when you’re close to or over 100% of your license limit and therefore at risk of experiencing drops.

Legacy licensing metrics

The following table explains metrics that might be present in your environment, but will be replaced by new metrics. The following metrics replace this table:

Persisted writes
Persisted cardinality
Matched writes

These metrics create the following tags during dashboard creation:

chronosphere_service

Metric name	Metric type	Description
`limit_service_cardinality_count` replaced by `chrono_metrics_persisted_cardinality_license_consumed`	Counter	Current cardinality count across all Collectors.
`limit_service_licensed_cardinality_limit` replaced by `chrono_metrics_persisted_cardinality_license_limit`	Counter	Current cardinality limit across all Collectors.
`limit_service_licensed_persist_limit` replaced by `chrono_metrics_persisted_writes_license_dpps_limit`	Counter	Current limit for data points persisted in the database across all Collectors, as defined in the contract.
`limit_service_capacity_limit`	Counter	Current capacity limit for data points persisted in the database across all Collectors, based on grant by Chronosphere.
`limit_service_persisted_count` replaced by `chrono_metrics_persisted_writes_license_dpps_consumed`	Counter	Total number of data points persisted in database.
`limit_service_matched_limit` replaced by `chrono_metrics_matched_writes_license_dpps_limit`	Counter	Current license limit for matched write DPPS by datapoint type.
`limit_service_capacity_limit` replaced by `chrono_metrics_matched_writes_license_dpps_capacity`	Counter	Current capacity limit for matched write DPPS by datapoint type.
`chronosphere_rule_metrics_matched` replaced by `chrono_metrics_matched_writes_license_dpps_consumed`	Counter	Consumption rate in DPPS of matched write license by datapoint type.

​Standard Metrics License

​Histogram Metrics License

​Capacity license metrics

​Matched writes

​Matched writes metrics

​Persisted writes

​Persisted writes metrics

​Persisted cardinality

​Persisted cardinality metrics

​How persisted cardinality limits work

​Manage persisted cardinality limits

​Avoid persisted cardinality limits

​Capacity limits

​Legacy licensing metrics

Standard Metrics License

Histogram Metrics License

Capacity license metrics

Matched writes

Matched writes metrics

Persisted writes

Persisted writes metrics

Persisted cardinality

Persisted cardinality metrics

How persisted cardinality limits work

Manage persisted cardinality limits

Avoid persisted cardinality limits

Capacity limits

Legacy licensing metrics