Alerting
Notifications
Notification policies

Notification policies

A notification policy connects a monitor with notifiers at warning and critical levels. The notification policy applies rules that define how to route an alert when it triggers, such as who to notify and through which notifier type. Chronosphere runs several checks to determine which notification policy to apply. You can define custom overrides for notification policies based on a signal's labels.

View notification policies

You can view available notification policies in the Chronosphere app, or return a list of notification policies using Chronoctl. You can view notification policies created in Terraform in the Chronosphere app, but can't modify them.

To view all notification policies:

  1. In the navigation menu select Alerts > Notification Policies.
  2. Use the search box to locate specific notification policies, or choose a notification policy from the Select a Team dropdown.
  3. Click a notification policy to view its definition.

Create a notification policy

Teams create and own notification policies, and each team can own multiple notification policies.

You can set a notification policy in these ways:

  • Per collection: If you configure a notification policy as the default policy on a collection, every monitor in that collection inherits the notification policy.
  • Per monitor: You can set a notification policy explicitly on each monitor rather than setting a default policy on a collection.

To create a notification policy:

  1. Go to Alerts > Notification Policies.
  2. On the Notification Policies page, click Create notification policy.
  3. Enter a name for the notification policy and select a team to own the policy.
  4. For both Critical Alert Notifiers and Warning Alert Notifiers:
    • Select the time period for resending notifications.
    • Choose a notifier to define who to notify and through which endpoint.
  5. Click Save.

Edit a notification policy

You can edit notification policies using the Chronosphere app, Chronoctl, or Terraform.

You can use the Chronosphere app to edit only notification policies created in the Chronosphere app.

To edit a notification policy:

  1. Go to Alerts > Notification Policies.
  2. Click the notification policy you want to edit.
  3. In the notification policy definition page, click Edit Policy.
  4. Make changes to your notification policy.
  5. Click Save.

Delete a notification policy

You can delete notification policies using the Chronosphere app, Chronoctl, or Terraform.

You can use the Chronosphere app to delete only notification policies created in the Chronosphere app.

To delete a notification policy:

  1. Go to Alerts > Notification Policies.
  2. Click the notification policy you want to delete. The notification policy definition page displays:
    1. Click Edit Policy.
    2. Click Delete notification policy.
  3. Click Delete to confirm that you want to delete the notification policy.

Override notification policy defaults

You can override notification policy defaults based on alert labels by adding an overrides collection to its definition. Use the alert_label_matchers collection to define the conditions for the override.

You can specify either a monitor label name or a signal label name as the value of the alert_label_matchers collection. If you specify any other label type, the override isn't processed for the specified monitor, even if the alerting series includes the specified label.

When an override matches the defined conditions, the notification defined in the specified notifiers collection triggers. If an override matches multiple conditions, only the first match triggers a notification. Any additional matches don't trigger a notification.

When a monitor has signal_per_series set to true (multiple alerts) as the signal grouping, you can use any label as a notification policy override.

To override a notification policy:

  1. Go to Alerts > Notification Policies.
  2. On the Notification Policies page, click Create notification policy.
  3. Enter a name for the notification policy and select a team to own the policy.
  4. For both Critical Alert Notifiers and Warning Alert Notifiers:
    • Select the time period for resending notifications.
    • Choose a notifier to define who to notify and through which endpoint.
  5. Click + Add Override Notifier to define a notification override:
    • Enter a label name and value to match on. To add another label and value to match on, click Add Matcher Label.
    • Define the conditions for both the Critical Alert Notifiers and Warning Alert Notifiers sections.
  6. Click Save.

Choose a notification policy

When a monitor triggers, Chronosphere runs checks to determine which notification policy to apply. First, Chronosphere checks the notification policy type to determine:

  • If the monitor has an explicit notification policy, use that policy.

  • If the associated collection has a default policy, use that policy. The notification_policy_slug or notification_policy_id attribute on a collection entity declares a default policy.

  • If the collection doesn't have a default policy but has an owned policy, use that policy. The bucket_slug or bucket_id attribute on a notification policy entity declares an owned policy.

    Owned policies are deprecated. Refer to migrate buckets to collections to migrate from buckets to collections.

From there, Chronosphere evaluates the alert severity:

  • If the alert severity is warning and matches a warning override, use the override notifier. Otherwise, use the default warning notifier.
  • If the alert severity is critical and matches a critical override, use the override notifier. Otherwise, use the default critical.

Chronosphere then delivers the alert using the selected notifier type.

WebhookSignals