Muting rules

Setting monitor muting rules

Use muting rules to mute notifications for an entire monitor, a signal from a monitor, or any stored time series. You can use this capability when you're fixing a known issue and don't want to keep receiving alerts, during scheduled maintenance windows, or when deploying new services.

View muting rules

To display a list of defined muting rules, in the navigation menu select Alerting > Muting rules.

You can filter the list of rules by choosing a rule status, which can be Active, Scheduled, or Expired.

Point to any muting rule to display additional actions:

  • Copy muting rule: Available for rules that are Active, Scheduled, or Expired.
  • Expire muting rule: Available for rules that are Active or Scheduled.

Select any muting rule in the list to view an overview, which includes the monitor or time series that the muting rule matches, when the muting rule starts and ends, who created the muting rule, and the total monitors and time series affected.

When viewing an active muting rule, you can display a list of alerts that the rule mutes during the specified time period. For example, you can show all muted alerts for the past hour to help determine whether the muting rule is effectively muting the intended alerts.

Active alerts are actively triggering, but are currently muted by the muting rule.

Resolved alerts were previously triggered and muted by the muting rule, but are no longer triggering.

When viewing an active muting rule, you can change the time range of the query to see which alerts are affected.

Add a muting rule

You can create a muting rule to mute an entire monitor or specific time series.

You can add a muting rule from the navigation menu from the list of defined monitors or directly from the list of muting rules.

To create a muting rule:

  1. Use one of these methods to start the process:

    • To mute an existing rule:
      1. In the navigation menu select Alerting > Monitors & Signals and click the monitor you want to mute.
      2. If you want to mute specific time series only (and not the entire monitor), choose the time series from the list in the Series Legend.
      3. On the same line as the monitor's title, click Mute.
    • To create an original muting rule, in the navigation menu select Alerting > Muting Rules, and click + Muting Rule.
  2. Define your muting rule, based on these settings:

    • Name: Required, but doesn't need to be unique.
    • What alerts should be muted?: Whether the muting rule should mute A Monitor or a Time Series, the Matcher which is the operator to match a value against a key, and the Key and Value to match on.
    • When should it start?: Start muting Now or at a specific Start time.
    • For how long?: The Duration or defined End time of the muting rule.

    Adding a muting rule from an existing monitor pre-populates the matching criteria without having to manually enter that information. As you define a muting rule, the Preview Alerts section displays how your muting rule impacts related alerts. Based on this preview, you can view and modify monitoring rule's results before the muting rule is active.

    Muting rules support regular expressions, which you can use to match key values against. Refer to the section about how to match on values using regular expressions.

  3. Click Save.

Extend a muting rule's duration

You can extend the duration of an active muting rule to increase the length of time that the muting rule remains active.

  1. On the Muting Rules page, click the muting rule whose duration you want to extend.
  2. On the Muting Rules Details page, click Edit.
  3. In the Extend end time by section, select Duration, and define the time period by which to extend the muting rule.
  4. Click Save.

Stop a muting rule

You can stop a muting rule, which deactivates the rule and changes its status to Expired. When viewing an expired muting rule, a counter displays the number of muted alerts and affected monitors while the rule was active.

  1. On the Muting Rules page, click the muting rule you want to stop.
  2. On the Muting Rules Details page, click Expire Muting Rule.
  3. In the confirmation dialog, click Confirm (Expire).

Delete a muting rule

Use the Chronoctl muting-rules delete command to permanently delete a muting rule. Specify the slug of the muting rule to delete as an argument. For example:

chronoctl muting-rules delete SLUG

Replace SLUG with the rule's slug, such as 3021f691-a0bf-4bdd-a4c8-f8468e5b272a.

Match on values using regular expressions

When creating a muting rule, you can use regular expressions to include or exclude values.

In the Chronosphere app, select either the PromQL match regex (=~) or does not match regex (!~) matchers, and then define regular expressions for the values you want to match on.

If you're creating a muting rule using Chronoctl, pass the -match-labels argument followed by a PromQL query that includes regular expressions.

For example, you might want to match on a time series with a label named envoy_cluster_name and include any values that begin with cluster_span_rc. When defining your muting rule in the Chronosphere app, select the match regex (=~) matcher, select envoy_cluster_name as the key, and enter cluster_span_rc.* as the value to match on. This regular expression pattern matches on values such as cluster_span_rc_shared__spanhandler_proxy_r-0. The equivalent command in Chronoctl is:

chronoctl muting-rules create --name "envoy mute" --match-labels '{envoy_cluster_name=~"cluster_span_rc.*"}' --ends-at +1h

After you create a muting rule in the Chronosphere app, select it from the list of muting rules and click Show JSON to view the complete JSON request for your muting rule. You can copy this JSON snippet and use it in a script with the Chronosphere API.

{
  "slug": "lifecycle-match-envoy_cluster_name",
  "name": "Muting Rule",
  "matchers": [
    {
      "type": "REGEXP_MATCHER_TYPE",
      "name": "envoy_cluster_name",
      "value": "cluster_span_rc.*"
    }
  ],
...
}