Notification policies

Notification policies

A notification policy connects a monitor with notifiers at warning and critical levels. The notification policy applies rules that define how to route an alert when it triggers, such as who to notify and through which notifier type. Chronosphere runs several checks to determine which notification policy to apply. You can define custom overrides for notification policies based on a signal's labels.

View notification policies

To view available notification policies, in Chronoctl use the notification-policies list command:

chronoctl notification-policies list

This returns a list of YAML documents, each representing a notification policy. For example, a Chronoctl listing of a notification policy in the example-team-name team might look like this:

api_version: v1/config
kind: NotificationPolicy
spec:
  slug: example-policy
  name: Example Policy
  created_at: "2022-02-16T05:55:30.000Z"
  updated_at: "2022-06-14T13:01:07.000Z"
  team_slug: example-team-name
  routes:
    defaults:
      warn: {}
      critical: {}
    overrides:
      - alert_label_matchers:
          - type: EXACT
            name: notify
            value: pinnacle-db-slack
          - type: REGEX
            name: notify
            value: ^(?:example.*|test.*|sample.*)$
        notifiers:
          warn:
            notifier_slugs:
              - example-1
          critical:
            notifier_slugs:
              - example-1

Chronoctl versions 0.46.0 and later let you apply this YAML output.

Create a notification policy

Teams create and own notification policies, and each team can own multiple notification policies.

You can set a notification policy in these ways:

  • Per collection: If you configure a notification policy as the default policy on a collection, every monitor in that collection inherits the notification policy.
  • Per monitor: You can set a notification policy explicitly on each monitor rather than setting a default policy on a collection.

To create a notification policy:

  1. Go to Alerting > Notification Policies.
  2. On the Notification Policies page, click + Add.
  3. Enter a name for the notification policy and select a team to own the policy.
  4. For both Critical Alert Notifiers and Warning Alert Notifiers:
    • Select the time period for resending notifications.
    • Choose a notifier to define who to notify and through which endpoint.
  5. Optional: To add a notifier override, click + Add Override Notifier and specify the key/value pair to match on and the notifier conditions to alert on.
  6. Click Save.

Override notification policy defaults

You can override notification policy defaults based on alert labels by adding an overrides collection to its definition. Use the alert_label_matchers collection to define the conditions for the override.

You can specify either a monitor label name or a signal group label name as the value of the alert_label_matchers collection. If you specify any other label type, the override isn't processed for the specified monitor, even if the alerting series includes the specified label.

When an override matches the defined conditions, the notification defined in the specified notifiers collection triggers. If an override matches multiple conditions, only the first match triggers a notification. Any additional matches don't trigger a notification.

When a monitor has signal_per_series set to true (multiple alerts) as the signal grouping, you can use any label as a notification policy override.

For example, the following YAML definition adds an override notifier when the component label has the value of mysql, and another if the importance label has the value low:

This example uses the type value EXACT for exact label matching. To use regular expression matching, provide the type value REGEX. These are the resource's only type values.

routes:
  defaults: 
  overrides:
    # If an alert has the component=mysql label, route to the db team.
    - alert_label_matchers:
        - type: EXACT,
          name: component
          value: mysql
      notifiers:
        critical:
          notifier_slugs:
            - db-team-urgent
        warn:
          notifier_slugs:
            - db-team
    - alert_label_matchers:
        - type: EXACT
          name: importance
          value: low
      notifiers:
        critical:
          notifier_slugs:
            - accounts-email

Choosing a notification policy

When a monitor triggers, Chronosphere runs checks to determine which notification policy to apply. First, Chronosphere checks the notification policy type to determine:

  • If the monitor has an explicit notification policy, use that policy.

  • If the associated collection has a default policy, use that policy. The notification_policy_slug or notification_policy_id attribute on a collection entity declares a default policy.

  • If the collection doesn't have a default policy but has an owned policy, use that policy. The bucket_slug or bucket_id attribute on a notification policy entity declares an owned policy.

    Owned policies are deprecated. Refer to migrate buckets to collections to migrate from buckets to collections.

From there, Chronosphere evaluates the alert severity:

  • If the alert severity is warning and matches a warning override, use the override notifier. Otherwise, use the default warning notifier.
  • If the alert severity is critical and matches a critical override, use the override notifier. Otherwise, use the default critical.

Chronosphere then delivers the alert using the selected notifier type.

Notification policy flow