Setting monitor muting rules

Use muting rules to mute notifications for an entire monitor, a signal from a monitor, or any stored time series. You can use this capability when you're fixing a known issue and don't want to keep receiving alerts, during scheduled maintenance windows, or when deploying new services.

View muting rules

Use the Chronosphere Observability Platform or Chronoctl to view a list of muting rules. Observability Platform provides detailed information about muting rules, such as which alerts a muting rule affects.

To display a list of defined muting rules, in the navigation menu select Alerts > Muting Rules.

You can take the following actions:

Filter the list of rules by choosing a rule status, which can be Active, Scheduled, or Expired.
Point to any muting rule to display additional actions:
- Copy muting rule: Available for rules that are Active, Scheduled, or Expired.
- Expire Muting Rule: Available for rules that are Active or Scheduled.
Select any muting rule in the list to view an overview, which includes the monitor or time series that the muting rule matches, when the muting rule starts and ends, who created the muting rule, and the total monitors and time series affected.

When viewing an active muting rule, you can display a list of alerts that the rule mutes during the specified time period. For example, you can show all muted alerts for the past hour to help determine whether the muting rule is effectively muting the intended alerts.

The Muted Alerts section displays the following muting rule statuses. You can change the time period to see which alerts the muting rule affects:

All alerts are a complete list of alerts muted during the selected time period.
Active alerts are actively triggering, but are currently muted by the muting rule.
Resolved alerts were previously triggered and muted by the muting rule, but are no longer triggering.

Create a muting rule

You can create a muting rule to mute an entire monitor or specific time series.

You can add a muting rule from the navigation menu from the list of defined monitors or directly from the list of muting rules.

Adding a muting rule from an existing monitor pre-populates the matching criteria without having to manually enter that information. As you define a muting rule, the Preview Alerts section displays how your muting rule impacts related alerts. Based on this preview, you can view and modify monitoring rule's results before the muting rule is active.

To create a muting rule:

Choose whether to mute an existing rule or create a new one:
- To mute an existing rule:
  1. In the navigation menu select Alerts > Monitors and click the monitor you want to mute.
  2. If you want to mute specific time series only (and not the entire monitor), choose the time series from the list in the Series Legend.
  3. On the same line as the monitor's title, click Mute.
- To create a new muting rule, in the navigation menu select Alerts > Muting Rules, and click Create muting rule.
Define your muting rule:
- Name: Required, but doesn't need to be unique.
- What alerts should be muted?: Whether the muting rule should mute A Monitor or a Time Series, the Matcher which is the operator to match a value against a key, and the Key and Value to match on.
- When should it start?: Start muting Now or at a specific Start time.
- For how long?: The Duration or defined End time of the muting rule.
  
  Muting rules support regular expressions, which you can use to match key values against. Refer to the section about how to match on values using regular expressions.
Click Save.

After you create a muting rule in Observability Platform, select it from the list of muting rules and click Show JSON to view the complete JSON request for your muting rule. You can copy this JSON snippet and use it in a script with the Chronosphere API.

{
  "slug": "lifecycle-match-envoy_cluster_name",
  "name": "Muting Rule",
  "matchers": [
    {
      "type": "REGEXP_MATCHER_TYPE",
      "name": "envoy_cluster_name",
      "value": "cluster_span_rc.*"
    }
  ],
...
}

Extend a muting rule's duration

You can extend the duration of an active muting rule to increase the length of time that the muting rule remains active. You can extend muting rules from Observability Platform only.

On the Muting Rules page, click the muting rule whose duration you want to extend.
On the Muting Rules Details page, click Edit.
In the Extend end time by section, select Duration, and define the time period by which to extend the muting rule.
Click Save.

Stop a muting rule

Use Observability Platform or Chronoctl to stop a muting rule, which deactivates the rule and changes its status to Expired.

When viewing an expired muting rule, a counter displays the number of muted alerts and affected monitors while the rule was active.

To stop a muting rule:

On the Muting Rules page, take one of the following actions:
- Hold the pointer over the muting rule you want to mute and click Expire Muting Rule.
- Click a muting rule to open the Muting Rules Details page, and then click Expire Muting Rule.
In the confirmation dialog, click Confirm (Expire).

Delete a muting rule

Select from the following methods to delete a muting rule.

Use the Chronoctl muting-rules delete command to permanently delete a muting rule. Specify the slug of the muting rule to delete as an argument. For example:

chronoctl muting-rules delete SLUG

Replace SLUG with the rule's slug, such as 3021f691-a0bf-4bdd-a4c8-f8468e5b272a.

Match on values using regular expressions

When creating a muting rule, you can use regular expressions to include or exclude values.

In Observability Platform, select either the PromQL match regex (=~) or does not match regex (!~) matchers, and then define regular expressions for the values you want to match on.

If you're creating a muting rule using Chronoctl, pass the -match-labels argument followed by a PromQL query that includes regular expressions.

For example, you might want to match on a time series with a label named envoy_cluster_name and include any values that begin with cluster_span_rc. You can use Observability Platform or Chronoctl to define a regular expression pattern that matches on values such as cluster_span_rc_shared__spanhandler_proxy_r-0.

When defining your muting rule in Observability Platform, specify the following criteria:

Select the match regex (=~) matcher.
Enter envoy_cluster_name as the key.
Enter cluster_span_rc.* as the value to match on.

Signals Alert metrics