OBSERVABILITY PLATFORM
Alerting
Notifications
Notification policies

Notification policies

A notification policy connects a monitor with notifiers at warning and critical levels. The notification policy applies rules that define how to route an alert when it triggers, such as who to notify and through which notifier type. Chronosphere Observability Platform runs several checks to determine which notification policy to apply. You can define custom overrides for notification policies based on a signal's labels.

View notification policies

Select from the following methods to view your available notification policies.

To view all notification policies:

  1. In the navigation menu select Alerts > Notification Policies.
  2. Use the search box to locate specific notification policies, or choose a notification policy from the Select a Team dropdown.
  3. Click a notification policy to view its definition.

The list includes notification policies created in Terraform, but you can't modify them.

Create a notification policy

You can set a notification policy in these ways:

  • Per collection: If you configure a notification policy as the default policy on a collection, every monitor in that collection inherits the notification policy.
  • Per monitor: You can set a notification policy explicitly on each monitor rather than setting a default policy on a collection.

The repeat interval is the frequency to resend alerts, which is 3600 seconds by default. If the repeat interval isn't set, or is set to 0, the value defaults to 3600.

The maximum repeat interval is 40 days. The repeat interval can not be lower than 300 seconds (five minutes), and should be a multiple of 300. Repeat intervals are rounded up to the next five minute increment.

You can configure a group_by field in notification policies both on the default and override levels. The value of group_by accepts a set of unique labels. For example, you can group by env for environments or by endpoint.

Grouping notifications with the group_by field on a notification policy is applied only if the associated monitor doesn't have any custom signals configured.

To create a notification policy:

  1. In the navigation menu select Alerts > Notification Policies.
  2. On the Notification Policies page, click Create notification policy.
  3. Enter a name for the notification policy. Optionally, select a team to own the policy.
  4. For both Critical Alert Notifiers and Warning Alert Notifiers:
    • Select the time period for resending notifications, which defaults to one hour.
    • Choose a notifier to define who to notify and through which endpoint.
  5. Click Save.

Edit a notification policy

Select from the following methods to edit notification policies.

You can use Observability Platform to edit only notification policies created in Observability Platform.

To edit a notification policy:

  1. In the navigation menu select Alerts > Notification Policies.
  2. Click the notification policy you want to edit.
  3. In the notification policy definition page, click Edit Policy.
  4. Make changes to your notification policy.
  5. Click Save.

Delete a notification policy

Select from the following methods to delete notification policies.

You can use Observability Platform to delete only notification policies created in Observability Platform.

To delete a notification policy:

  1. In the navigation menu select Alerts > Notification Policies.
  2. Click the notification policy you want to delete. The notification policy definition page displays:
    1. Click Edit Policy.
    2. Click Delete notification policy.
  3. Click Delete to confirm that you want to delete the notification policy.

Override notification policy defaults

You can override notification policy defaults based on alert labels by adding an overrides collection to its definition. Use the alert_label_matchers collection to define the conditions for the override.

You can specify either a monitor label name or a signal label name as the value of the alert_label_matchers collection. If you specify any other label type, the override isn't processed for the specified monitor, even if the alerting series includes the specified label.

When an override matches the defined conditions, the notification defined in the specified notifiers collection triggers. If an override matches multiple conditions, only the first match triggers a notification. Any additional matches don't trigger a notification.

When a monitor has signal_per_series set to true (multiple alerts) as the signal grouping, you can use any label as a notification policy override.

To override a notification policy:

  1. In the navigation menu select Alerts > Notification Policies.
  2. On the Notification Policies page, click Create notification policy.
  3. Enter a name for the notification policy. Optionally, select a team to own the policy.
  4. For both Critical Alert Notifiers and Warning Alert Notifiers:
    • Select the time period for resending notifications.
    • Choose a notifier to define who to notify and through which endpoint.
  5. Click + Add Override Notifier to define a notification override:
    • Enter a label name and value to match on. To add another label and value to match on, click Add Matcher Label.
    • Define the conditions for both the Critical Alert Notifiers and Warning Alert Notifiers sections.
  6. Click Save.

Choose a notification policy

When a monitor triggers, Observability Platform runs checks to determine which notification policy to apply. First, Observability Platform checks the notification policy type to determine:

  • If the monitor has an explicit notification policy, use that policy.

  • If the associated collection has a default policy, use that policy. The notification_policy_slug or notification_policy_id attribute on a collection entity declares a default policy.

  • If the collection doesn't have a default policy but has an owned policy, use that policy. The bucket_slug attribute on a notification policy entity declares an owned policy.

    Owned policies are deprecated. Refer to migrate buckets to collections to migrate from buckets to collections.

From there, Observability Platform evaluates the alert severity:

  • If the alert severity is warning and matches a warning override, use the override notifier. Otherwise, use the default warning notifier.
  • If the alert severity is critical and matches a critical override, use the override notifier. Otherwise, use the default critical.

Observability Platform then delivers the alert using the selected notifier type.

WebhookSignals