Drop rules
Based on your amount of available storage and rate limits, you might want to reduce the amount of metrics that you continue to store so that you’re keeping only those metrics you need. For example, using Telemetry Usage Analyzer, you might find a metric has a low utilization score. Metrics with high volume and low utilization are candidates for dropping.
Use drop rules to omit incoming metrics based on labels, and have Chronosphere Observability Platform persist only those metrics you need. Define filters to specify the metric or label names you want to drop. You can also drop metrics based on a data point value.
The Collector scrapes data from an app, and sends the data to Chronosphere where you can apply drop rules. Any metrics that weren’t impacted by drop rules are then aggregated in Observability Platform.
Drop rules affect data from the point of rule creation onward. When you query for the metric:
- Data can display in autocomplete searches for a period of approximately two hours.
- Queries covering the time period before the drop rule took effect display the metric until the retention period expires.
View and filter created drop rules
To view the drop rules defined on your instance, you must have administrative privileges.
Select from the following methods to view your available drop rules.
To view drop rules:
- In the navigation menu, click Go to Admin and then select Control > Drop Rules.
- Use the Search rules text box to find a specific rule. This search looks for any text in any rule.
- Select a value from the Status menu to filter for rules matching the selected status.
- Select a drop rule from the list that you want to view.
- Optional: To annotate a drop rule with additional context, the Edit Drop Rule panel click the Comments tab to add a comment.
For information about viewing, copying, or downloading rule configurations, see Rule configuration.
Create a drop rule
Use drop rules to drop the ingestion of certain metrics that you don’t need to persist. For example, dropping metrics from a specific cluster, or test data.
Drop rules can have multiple filter conditions on different labels, making it possible to drop a subset of the series matching a particular metric name. Use multiple filters for cases where only a subset of a metric’s data is used. For example, if a metric is used in an alert but examines only series related to error statuses, you can drop non-error status series.
You can use glob syntax in your drop rule to match
specific values. Use the value_based_drop
object to define a
value-based drop rule that drops metrics based on
a target data point value.
Drop rules have these definition limitations:
- The rule can’t be longer than 8,192 characters.
- A drop rule
slug
must contain only alphanumeric characters, dashes, or underscores. The allowed regular expression for matching is[a-zA-Z0-9-_]
. Spaces aren’t allowed. - Each rule
name
must be unique.
After creating a drop rule, it takes roughly one minute for the rule to take effect. Wait at least one minute after creating a drop rule before making updates to it.
A metric that matches any drop rule is dropped, regardless of overlap with drop rules
created using other methods. For example, if a metric matches a drop rule created
with Terraform that also matches a subset of a drop rule created with the
CreateDropRule
endpoint,
the metric is dropped.
A drop rule containing kubernetes_cluster:dev
matches and drops lower value metrics
containing this key/value pair. To exclude specific values that also match this rule,
use a single filter
with an AND
logical operator to specify a “does not equal”
match. The following drop rule matches any metric containing
kubernetes_cluster:dev
, but not metrics that also contain the specified
container_name
values:
"kubernetes_cluster:dev AND container_name:!{log_service|data_service}"
Select from the following methods to create a drop rule.
Users can modify Terraform-managed resources only by using Terraform. Learn more.
To create a drop rule, you must have administrative privileges:
-
In the navigation menu, click Go to Admin and then select Control > Drop Rules.
-
Click Create drop rule.
-
In the Create drop rule drawer, select a Status for the drop rule, which can be one of the following values:
- Enabled: Create the rule and make it active immediately.
- Preview: Create the rule and evaluate it, but don’t drop data.
- Disabled: Create the rule, but don’t evaluate it to drop data.
-
Enter a Name for the rule.
-
Enter one or more Matchers, which are labels with or without a specific value. You can use glob syntax in the Value field.
To specify multiple values, separate values with a comma and surround them with curly braces (
{}
). For example,key={value1,value2,value3}
.To match values containing a comma, specify the values without braces. For example,
key=value,continue
.When allowing only particular metrics, you might need a double negation, which requires specific logical filtering:
keep: a==A && b==B drop: !(a==A && b==B)
In combination, this logic leads to a drop rule where
!a==A || !b==B --> a!=A || b!=B
.Comma-separated lists can’t include spaces after commas. Rules with a space after a comma don’t evaluate.
-
To define a value-based drop rule, choose 0 or 1 in the Select drop value menu.
-
Click Review.
-
Review the rule to ensure it’s correct.
-
Select the checkbox for I have reviewed the proposed changes and they look correct.
-
Click Save rule to save the rule, or click Back to change the rule.
Chronoctl example
The following resource defines a drop rule named Drop node exporter
in PREVIEW
mode, with the slug drop-node-exporter-metrics
:
api_version: v1/config
kind: DropRule
spec:
name: "Drop node exporter"
# Unique identifier for rule
slug: "drop-node-exporter-metrics"
filters:
- name: job
value_glob: node_exporter
# Boolean to determine if rule is currently active
mode: PREVIEW
value_based_drop:
enabled: true
# The target data point value at which to drop metrics.
target_drop_value: 75
# Boolean to determine whether to drop data points that are NaN.
drop_nan_value: "true"
Terraform example
The following definition creates a drop rule that Terraform refers to as
drop-node-exporter
, with the name Drop node exporter
. The rule drops all metrics
from the node_exporter
job.
resource "chronosphere_drop_rule" "drop-node-exporter" {
# Name for the rule
name = "Drop node exporter"
# Unique identifier for rule; generated automatically if left blank
slug = "drop-node-exporter-metrics"
# Label matcher to match metrics that should be dropped; all labels must match
query = [
"job=node_exporter",
]
# Boolean to determine if rule is currently active
active = true
# Boolean to determine whether to drop data points that are NaN.
drop_nan_value = true
}
This rule is an example of a drop rule based on the value of the incoming data point:
resource "chronosphere_drop_rule" "value-based-drop-rule" {
# Name for the rule
name = "Drop Rule"
# Label matcher to match metrics that should be dropped; all labels must match
query = [
"ab=123",
]
# Boolean to determine if the rule is currently active
active = true
# Configuration of which metric values to drop; must be 0 or 1
value_based_drop {
target_drop_value = 1
}
# Boolean to determine whether to drop data points if the values are NaN
drop_nan_value = true;
}
Matching with glob syntax
Drop rules support glob syntax, but don’t support PromQL operators in the
value_glob
field to indicate negation. The following glob syntax applies to drop
rules:
-
To match on any value, include an asterisk (
*
) in thevalue_glob
field. For example, the following rule matches values that start withenvoy_cluster_grpc
for the__name__
metricfilters: - name: __name__ value_glob: envoy_cluster_grpc*
-
To not match on a value for a metric label, include an exclamation mark (
!
) in thename
field. For example, the following rules match any metrics that don’t have acontainer
label with a value ofdatabase-backup
ordatabase-restore
:filters: - name: container! value_glob: database-{backup,restore}
Verify your glob syntax with Live Telemetry Analyzer to ensure your query matches the correct metrics.
You can also create drop rules in preview mode to evaluate a rule without dropping data. This capability lets you test a rule without impacting your persisted data.
Glob syntax examples
The following definition for a drop rule in Chronoctl includes a few different rules
that include matching on any value (*
) and negation syntax (!
):
- Drops any metric that doesn’t have an
id
field with a forward slash (/
) as a value. - Drops any metric that includes the
pod
key, with any value. - Drops any metric that doesn’t have the
container
field, with any value.
{
"drop_rule": {
"slug": "cAdvisor-pause-containers",
"name": "cAdvisor pause containers",
"mode": 1,
"filters": [
{
"name": "job",
"value_glob": "k8s-cadvisor"
},
{
"name": "id!",
"value_glob": "/"
},
{
"name": "pod",
"value_glob": "*"
},
{
"name": "container!",
"value_glob": "*"
},
]
}
}
The following drop rule for the
CreateDropRule API drops any
metric containing the service
label with a value of api
that also has a cluster
label that begins with dev
.
{
"drop_rule": {
"slug": "api-drop-rule",
"name": "api drop rule",
"mode": 1,
"filters": [
{
"name": "service",
"value_glob": "api"
}
{
"name": "cluster",
"value_glob": "dev*"
}
]
}
}
To keep only metrics with a specific label value, use glob syntax to specify a “does
not equals” match with an exclamation mark !
in the name
field to match specified
values.
For example, the following drop rule drops any metric containing the
service
label with a value of api
, that doesn’t also have an env
label with a
value of production
. This drop rule also applies if the env
label isn’t set.
{
"drop_rule": {
"slug": "api-drop-rule",
"name": "api drop rule",
"mode": 1,
"filters": [
{
"name": "service",
"value_glob": "api"
}
{
"name": "env!",
"value_glob": "production"
}
]
}
}
A label with an empty value is the same as a label that isn’t present.
Use an asterisk *
in the value_glob
field to match metrics with a particular
label, regardless of the value. Conversely, to drop only metrics that don’t have a
particular label, use an exclamation mark !
in the name
field to specify a “does
not equal” match, in conjunction with an asterisk *
to match all values.
For example, the following drop rule drops any metric containing the service
label
with a value of api
that doesn’t also have an env
label.
{
"drop_rule": {
"slug": "api-drop-rule",
"name": "api drop rule",
"mode": 1,
"filters": [
{
"name": "service",
"value_glob": "api"
}
{
"name": "env!",
"value_glob": "*"
}
]
}
}
Observability Platform glob syntax doesn’t support using two asterisks where one of
them is in the middle of a string. For example, *k8s*staging
isn’t valid.
Define a value-based drop rule
Use the value_based_drop
object to define a value-based drop rule. The
target_drop_value
parameter specifies the target data point value at which to drop
metrics.
The value_based_drop
parameter drops time series based on data point values of 0
or 1
. This type of drop rule can significantly reduce data points per second, but
changes metrics from continuous series to sparse metrics. Adding a rule of this type
can change the behavior of existing monitors or dashboards that use the target
metric.
You can include the drop_nan_value
parameter to drop data point values that are Not
a Number (NaN). If set to true
, Chronosphere drops NaN data points, along with any
published
staleness markers.
If you set the drop_nan_value
parameter to true
in conjunction with the
value_based_drop
parameter, Observability Platform drops data points matching the
value or any NaN values.
If you want to drop only NaN values, set drop_nan_value
to true
without a
corresponding value_based_drop
parameter. When no value is set for
drop_nan_value
, all values are dropped, including NaN values.
View drop rule impact
After creating a drop rule, query the
chrono_metrics_drop_rule_dpps_matched
metric to understand the impact of your drop rule. Use the labels this metric
provides to answer these questions:
- Which data points were dropped by active drop rules?
- Which active rules overlap or duplicate other drop rules?
- What are the total matches across all rules?
To understand the impact of a drop rule:
-
In the navigation menu select Explorers > Metrics Explorer.
-
In the Metrics Explorer page, click Builder and then in the Metric field, enter the following metric name:
chrono_metrics_drop_rule_dpps_matched
-
Click Run to run the query.
The results include all drop rules.
-
Click the Filters field. In the Labels subfield, select
slug
as the label to match on, and then click the Value subfield to search for the name of your drop rule. -
Click Run to run the updated query and narrow the results.
-
In the Aggregation field, select Sum as the aggregation, and then select applied and mode as the values to aggregate by.
-
Click Run to run the updated query.
You can add filters to narrow the results to answer specific questions. The following examples provide some common questions, and include the query to run in Metrics Explorer to answer each question:
-
How can I preview how much data a given drop rule will drop?
sum(chrono_metrics_drop_rule_dpps_matched{mode="preview", slug="SLUG"})
Replace
SLUG
with the slug of your drop rule. This metric creates the drop rule inpreview
mode. You can then use the following metric to enable the drop rule and apply it. -
How much data does an existing, enabled rule drop?
sum(chrono_metrics_drop_rule_dpps_matched{mode="enabled", applied="true", slug="SLUG"})
Replace
SLUG
with the slug of your drop rule. -
How much data am I actually dropping across my existing drop rules?
sum by (slug) (chrono_metrics_drop_rule_dpps_matched{mode="enabled", applied="true"})
-
What’s the overall combined number of data points dropped as a result of my enabled and preview rules?
sum(chrono_metrics_drop_rule_dpps_matched{applied="true"})
-
Which rules aren’t effective because they’re duplicates of existing rules that are dropping data?
sum by (slug) (chrono_metrics_drop_rule_dpps_matched{mode="enabled", applied="false"})
Disable a drop rule
Select from the following methods to disable a drop rule.
Users can modify Terraform-managed resources only by using Terraform. Learn more.
To disable an existing drop rule in Observability Platform, you must have administrative privileges:
- In the navigation menu, click Go to Admin and then select Control > Drop Rules.
- Select the rule you want to disable from the list.
- Toggle the rule to the Disabled state.
- Click Save.
To re-enable the rule, use the previous steps and instead toggle Always Active.
Delete a drop rule
Select from the following methods to delete a drop rule.
- In the navigation menu select Shaping > Drop Rules.
- Select the rule you want to delete from the list.
- Click the delete icon to delete the rule.