Drop rules
Based on your amount of available storage and rate limits, you might want to reduce the amount of metrics that you continue to store so that you're keeping only those metrics you need. For example, using Telemetry Usage Analyzer, you might find a metric has a low utilization score. Metrics with high volume and low utilization are candidates for dropping.
Use drop rules to omit incoming metrics based on labels, and have Chronosphere Observability Platform persist only those metrics you need. Define filters to specify the metric or label names you want to drop. You can also drop metrics based on a data point value.
The Collector scrapes data from an app, applies drop rules, and then publishes data to Chronosphere for aggregation.
Drop rules affect data from the point of rule creation onward. When you query for the metric:
- Data can display in autocomplete searches for a period of approximately two hours.
- Queries covering the time period before the drop rule took effect display the metric until the retention period expires.
Define a value-based drop rule
Use the value_based_drop object to define a value-based drop rule. The
target_drop_value parameter specifies the target data point value at which to drop
metrics.
The value_based_drop parameter drops time series based on data point values of 0
or 1. This type of drop rule can significantly reduce data points per second, but
changes metrics from continuous series to sparse metrics. Adding a rule of this type
can change the behavior of existing monitors or dashboards that use the target
metric.
You can include the drop_nan_value parameter to drop data point values that are Not
a Number (NaN). If set to true, Chronosphere drops NaN data points, along with any
published
staleness markers.
If you set the drop_nan_value parameter to true in conjunction with the
value_based_drop parameter, Observability Platform drops data points matching the
value, or drops any NaN values. If you want to drop only NaN values, set
drop_nan_value to true without a corresponding value_based_drop parameter. When
no value is set fordrop_nan_value, all values are dropped, including NaN values.
View and filter created drop rules
To view the drop rules defined on your instance, you must have administrative privileges.
Select from the following methods to view your available drop rules.
To view drop rules:
-
In the navigation menu, click Go to Admin and then select Control > Drop Rules.
-
Use the Search rules text box to find a specific rule. This search looks for any text in any rule.
-
Select a value from the Status menu to filter for rules matching the selected status.
For information about viewing, copying, or downloading rule configurations, see Rule configuration.
Create a drop rule
Use drop rules to drop the ingestion of certain metrics that you don't need to persist. For example, dropping metrics from a specific cluster, or test data.
Drop rules can have multiple filter conditions on different labels, making it possible to drop a subset of the series matching a particular metric name. Use multiple filters for cases where only a subset of a metric's data is used. For example, if a metric is used in an alert but examines only series related to error statuses, you can drop non-error status series.
Drop rules have these definition limitations:
- The rule can't be longer than 4,096 characters.
- A drop rule slug must contain only alphanumeric characters, dashes, or underscores.
The allowed regular expression for matching is
[a-zA-Z0-9-_]. Spaces aren't allowed. - Each rule name must be unique.
Verify your glob syntax with Live Telemetry Analyzer to ensure your query matches the correct metrics.
Observability Platform glob syntax doesn't support using two asterisks where one of
them is in the middle of a string. For example, *k8s*staging isn't valid.
Select from the following methods to create a drop rule.
Users cannot modify Terraform-managed resources in the user interface, with Chronoctl, or by using the API. Learn more.
To create a drop rule, you must have administrative privileges:
-
In the navigation menu, click Go to Admin and then select Control > Drop Rules.
-
Click + Drop Rule.
The drop rule consists of the following data model:
-
A Name for the rule.
-
A list of one or more Matchers, which are labels (a Key) with or without a specific value. You can use glob syntax in the Value field.
To specify multiple values, separate values with a comma and surround them with curly braces (
{}). For example,key={value1,value2,value3}.To match values containing a comma, specify the values without braces. For example,
key=value,continue.When allowing only particular metrics, you might need a double negation, which requires specific logical filtering:
keep: a==A && b==B drop: !(a==A && b==B)In combination, this logic leads to a drop rule where
!a==A || !b==B --> a!=A || b!=B.Comma-separated lists can't include spaces after commas. Rules with a space after a comma don't evaluate.
-
-
Click Review.
-
Review the rule to ensure it's correct.
-
Select the checkbox for I have reviewed the proposed changes and they look correct.
-
Click Save to finish, or Back to Edit to change the rule.
Disable a drop rule
Select from the following methods to disable a drop rule.
Users cannot modify Terraform-managed resources in the user interface, with Chronoctl, or by using the API. Learn more.
To disable an existing drop rule in Observability Platform, you must have administrative privileges:
- In the navigation menu, click Go to Admin and then select Control > Drop Rules.
- Select the rule you want to disable from the list.
- Toggle the rule to the Disabled state.
- Click Save.
To re-enable the rule, use the previous steps and instead toggle Always Active.
Delete a drop rule
Select from the following methods to delete a drop rule.
- In the navigation menu select Shaping > Drop Rules.
- Select the rule you want to delete from the list.
- Click the delete icon to delete the rule.