Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt

Use this file to discover all available pages before exploring further.

Chronosphere uses the following terms when describing licensing concepts and usage in Chronosphere Observability Platform. To take action and protect against overspending, configure budgets. For each budget, define thresholds and priorities to define the actions to take when a threshold is exceeded, and control the data that gets dropped. You can then attach budgets to a partition. In a capacity model, quotas determine which data drops first. You can split the total system-persisted writes per second into per-pool quota allocations.

​
Metric license types

Observability Platform defines two types of metric licenses: the Standard Metrics License and Histogram Metrics License.

​
Standard Metrics License

The Standard Metrics License measures aggregations, persisted writes, and persisted cardinality license consumption for the following Observability Platform metric types:
  • Cumulative counter
  • Delta counter
  • Gauge
Because Observability Platform aggregates and persists legacy Prometheus histograms and OpenTelemetry explicit bucket layout histograms as cumulative or delta counters, these metrics consume Standard Metrics License capacity.

​
Histogram Metrics License

The Observability Platform histogram metric type supports both OpenTelemetry exponential histograms and Prometheus native histograms. The Histogram Metrics License measures aggregations, persisted writes, and persisted cardinality license consumption for the following Observability Platform metric types:
  • Cumulative exponential histogram
  • Delta exponential histogram
Use the Trends tab of the License Overview page to observe histogram persisted writes, matched writes, and persisted cardinality in the Metrics consumption trends graph.

​
Matched writes

Matched writes are the number of writes per second being matched for transformation and reshaping by the Observability Platform aggregation tier. A matched write is counted for each data point matched into each aggregator rule, whether rollup or downsampling. If a data point matches one rule, that’s one matched write. If a data point matches two rules, that’s two matched writes. The sum of the matched data points per second per rule equals the total matched writes. Recording rules aren’t considered an aggregation rule for the purpose of counting matched writes. Writes also depend on your Collector scrape interval. Increasing the scrape interval produces fewer writes, but can reduce visibility. See your current Matched Writes level in the License Overview Snapshot in the Metrics Consumption section. On the Trends page, review usage over time in the Metrics Consumption Trends graph.

​
Matched writes metrics

The following metrics apply to matched writes, which are the number of writes per second being matched for transformation and reshaping by the Observability Platform aggregation tier.
Metric nameDescription
chrono_metrics_matched_writes_license_dpps_limitLicense limit for matched write DPPS by datapoint type.
chrono_metrics_matched_writes_license_dpps_capacityCapacity limit for matched write DPPS by datapoint type.
chrono_metrics_matched_writes_license_dpps_consumedConsumption rate in DPPS of matched write license by datapoint type.
Query the following metric to understand if data is actively being dropped: 
chrono_metrics_matched_license_dpps_dropped

​
Persisted writes

The number of persisted writes to the Observability Platform database consists of the following: 
(Number of unaggregated, raw data points written to the database)
+ (Number of aggregated data points written to the database)
If you exceed 100% of your persisted writes capacity limit, data points will be dropped before being persisted. To improve and enhance performance, stability, and features, Observability Platform adds a small number of additional time series to your database. These data points aren’t counted against your license. You can review your current usage in the Persisted writes graph on the License Overview Snapshot page, in the Metrics consumption section. To see changes over time, select Trends, and review the Metrics consumption trends graph.

​
Persisted writes metrics

The following metrics apply to persisted writes, which are writes to the Observability Platform database.
Metric nameDescription
chrono_metrics_persisted_writes_license_dpps_limitLicense limit for persisted write DPPS by datapoint type.
chrono_metrics_persisted_writes_license_dpps_capacityCapacity limit for persisted write DPPS by datapoint type.
chrono_metrics_persisted_writes_license_dpps_consumedConsumption rate in DPPS of persisted write license by datapoint type.
For each of these metrics, you can query by datapoint_type, such as histogram or standard. For the _dpps_capacity and _dpps_consumed metrics, you can additionally query by pool_name, and priority. For example, the following query returns the consumption rate in DPPS of your persisted write license for histogram data points on the Auth Services pool. Because priority isn’t specified, the query returns a series for each priority: 
chrono_metrics_persisted_writes_license_dpps_consumed{datapoint_type="histogram", pool_name="Auth Services"}
If you wanted to return only high priority histogram data points for the Auth Services pool, specify priority = "high" in your query: 
chrono_metrics_persisted_writes_license_dpps_consumed{datapoint_type="histogram", pool_name="Auth Services", priority="high"}
Query the following metric to understand if data is actively being dropped: 
chrono_metrics_persisted_license_dpps_dropped{limit="persisted_writes"}

​
Persisted cardinality

Matched writes and persisted writes measure the rate of data points per second at any given moment in time. Persisted cardinality operates differently, because it’s a cumulative measure that calculates the count of the unique time series of the persisted writes that Observability Platform stores, seen over the last 2.5 hours. This measure is also known as Active Time Series (ATS). Persisted cardinality can be influenced by a change in ingested metrics, or you can use rollup rules to downsample and aggregate metrics before they’re stored. Because this measure is cumulative, reductions in persisted cardinality aren’t reflected immediately, as inactive time series continue counting until they fall outside the 2.5 hour rolling window. Read more about persisted cardinality limits work and how to manage them: To see persisted cardinality license usage changes over time, in the License overview select Trends, and review the Metrics consumption trends graph.

​
Persisted cardinality metrics

The following metrics apply to persisted cardinality. This is a cumulative measure that calculates the count of the unique time series of the persisted writes that Observability Platform stores, as seen over the last 2.5 hours only.
Metric nameDescription
chrono_metrics_persisted_cardinality_license_limitLicense limit for active persisted time series cardinality.
chrono_metrics_persisted_cardinality_license_capacityCapacity limit for active persisted time series cardinality.
chrono_metrics_persisted_cardinality_license_consumedConsumption of the persisted write cardinality limit by datapoint type.
Query the following metric to understand if data is actively being dropped: 
chrono_metrics_persisted_license_dpps_dropped{limit="persisted_cardinality"}

​
How persisted cardinality limits work

Persisted cardinality is comparable to a leaky bucket. Over time, new series can be added until the bucket is full. When the bucket is at maximum capacity, there’s no space for new time series, so they’re rejected. When existing time series expire, they make room for new series. In the following example, the persisted cardinality capacity is five unique time series. The animated image shows the lifecycle of six, unique time series (A, B, C, D, E, and F) as new data points are added, and as other data points expire.
Animated image showing data points being introduced. When the persisted cardinality limit is reached, no more time series are accepted.
As data points are introduced, they’re either accepted or rejected based on whether the persisted cardinality bucket is full (reached maximum capacity), and whether the related time series already exists in the bucket:
  • If the bucket is at maximum capacity and the series already exists, the data point is accepted.
  • If the bucket is at maximum capacity and the series doesn’t exist, the series is rejected.
The following table shows how data points A3, E1, and F1 are processed, based on the bucket status:
Data pointStatusDescription
A3Time series A is in the bucket, so data point A3 is accepted.
E1Time series E isn’t in the bucket, but the bucket has space for one more time series, so data point E1 is accepted in time series E.
F1Time series F isn’t in the bucket, and the bucket is at capacity, so data point F1 is rejected.
Over time, data points expire based on when they entered the bucket. When data points exceed the 2.5 hour window, they’re excluded from the persisted cardinality bucket. In the example, data points A1 and D1 expired, so they’re excluded from the bucket. When data point C1 expires, it’s also excluded. Because data point C1 is the last data point in time series C, the entire series is removed, making space for a new time series in the bucket.
Data pointStatusDescription
A1Data point A1 expired, so it’s excluded from the bucket.
D1Data point A1 expired, so it’s excluded from the bucket.
C1Data point C1 is expiring, so it’s excluded from the bucket.

​
Manage persisted cardinality limits

If your organization exceeds 100% of their Persisted Cardinality Capacity Limit, data points for any new time series not seen in the last 2.5 hours will be dropped until you’re under this limit. Data points for existing time series will continue to be persisted. Series that are more stable or regularly emitted aren’t at risk of being dropped because they’re always in the system, and aren’t categorized as new series. For example, series that don’t change any labels are considered more stable. To fully resolve a penalty period, the rate of new series must be less than the rate of expiring series. The higher the differential between these rates, the faster the penalty resolves. To manage persisted cardinality limits:
  • Review the Persisted Cardinality Quotas dashboard, the Usage Dashboard and the Metric Growth dashboard to understand the source of cardinality growth.
  • Create drop rules and aggregation rules like mapping rules and rollup rules to roll away sources of growth. Old series remain in the cardinality window for 2.5 hrs.
  • Use the Recommendations page to help identify metrics and labels with no usage or utility over the past 30 days. You can then create drop rules and rollup rules based on the recommendations.
The 2.5 hour expiration window is a rolling window, which means the constant rate of expiring series makes room for an equal rate of new series to be added. This behavior means the penalty period you experience can be much shorter than 2.5 hours. Queries with time ranges that overlap with penalty periods can produce inconsistent or partial results due to the dropping of time series during those periods.

​
Avoid persisted cardinality limits

Create thresholds on budgets to help manage both anomalous spikes in data and slow data growth over time. Configure actions on each threshold and set a priority to determine what data to drop, and in what order. Use the following tools and techniques to avoid hitting persisted cardinality limits:
  • Review the Persisted Cardinality Quotas dashboard, the Usage Dashboard and the Metric Growth dashboard to understand the source of cardinality growth.
  • Learn about different methods to reduce cardinality.
  • Proactively create drop rules and aggregation rules like mapping rules and rollup rules ahead of potential overages to evict older time series and make room for new ones.
  • Proactively define thresholds on budgets to better manage persisted cardinality.
  • If your organization knows which new metrics services are generating, try to control the rate that new series are introduced through smaller, more incremental deploys.

​
Capacity limits

Capacity limits only apply to capacity pricing. If your organization uses the consumption model, see Manage consumption.
Capacity limits indicate your maximum license capacity for telemetry data in Observability Platform. Exceeding your capacity limits incurs penalties, which can result in dropped data. Dropped data can affect dashboards, alerts, and other reports. Licensing capacity is based on your telemetry types and usage. The license limit indicates your contractual license with Chronosphere. The capacity and license limits display in the Contracts tab of the License Overview page. These limits are broken down into individual limit graphs:
  • Persisted writes
  • Matched writes
  • Persisted cardinality
  • Histogram persisted writes
  • Histogram matched writes
  • Histogram persisted cardinality
Chronosphere recommends creating alerts using the existing capacity limit metrics, which are also used in the License Overview. Use alerts to be notified when you’re close to or over 100% of your license limit and therefore at risk of experiencing drops.

​
Legacy licensing metrics

The following table explains metrics that might be present in your environment, but will be replaced by new metrics. The following metrics replace this table: These metrics create the following tags during dashboard creation:
  • chronosphere_service
Metric nameMetric typeDescription
limit_service_cardinality_count
replaced by
chrono_metrics_persisted_cardinality_license_consumed		CounterCurrent cardinality count across all Collectors.
limit_service_licensed_cardinality_limit
replaced by
chrono_metrics_persisted_cardinality_license_limit		CounterCurrent cardinality limit across all Collectors.
limit_service_licensed_persist_limit
replaced by
chrono_metrics_persisted_writes_license_dpps_limit		CounterCurrent limit for data points persisted in the database across all Collectors, as defined in the contract.
limit_service_capacity_limitCounterCurrent capacity limit for data points persisted in the database across all Collectors, based on grant by Chronosphere.
limit_service_persisted_count
replaced by
chrono_metrics_persisted_writes_license_dpps_consumed		CounterTotal number of data points persisted in database.
limit_service_matched_limit
replaced by
chrono_metrics_matched_writes_license_dpps_limit		CounterCurrent license limit for matched write DPPS by datapoint type.
limit_service_capacity_limit
replaced by
chrono_metrics_matched_writes_license_dpps_capacity		CounterCurrent capacity limit for matched write DPPS by datapoint type.
chronosphere_rule_metrics_matched
replaced by
chrono_metrics_matched_writes_license_dpps_consumed		CounterConsumption rate in DPPS of matched write license by datapoint type.