Notification policies
A notification policy connects a monitor with notifiers at warning and critical levels. The notification policy applies rules that define how to route an alert when it triggers, such as who to notify and through which notifier type. Chronosphere runs several checks to determine which notification policy to apply. You can define custom overrides for notification policies based on a signal's labels.
View notification policies
You can view available notification policies in the Chronosphere app, or return a list of notification policies using Chronoctl. You can view notification policies created in Terraform in the Chronosphere app, but can't modify them.
To view all notification policies:
- In the navigation menu select Alerts > Notification Policies.
- Use the search box to locate specific notification policies, or choose a notification policy from the Select a Team dropdown.
- Click a notification policy to view its definition.
Create a notification policy
Teams create and own notification policies, and each team can own multiple notification policies.
You can set a notification policy in these ways:
- Per collection: If you configure a notification policy as the default policy on a collection, every monitor in that collection inherits the notification policy.
- Per monitor: You can set a notification policy explicitly on each monitor rather than setting a default policy on a collection.
To create a notification policy:
- Go to Alerts > Notification Policies.
- On the Notification Policies page, click Create notification policy.
- Enter a name for the notification policy and select a team to own the policy.
- For both Critical Alert Notifiers and Warning Alert Notifiers:
- Select the time period for resending notifications.
- Choose a notifier to define who to notify and through which endpoint.
- Click Save.
Edit a notification policy
You can edit notification policies using the Chronosphere app, Chronoctl, or Terraform.
You can use the Chronosphere app to edit only notification policies created in the Chronosphere app.
To edit a notification policy:
- Go to Alerts > Notification Policies.
- Click the notification policy you want to edit.
- In the notification policy definition page, click Edit Policy.
- Make changes to your notification policy.
- Click Save.
Delete a notification policy
You can delete notification policies using the Chronosphere app, Chronoctl, or Terraform.
You can use the Chronosphere app to delete only notification policies created in the Chronosphere app.
To delete a notification policy:
- Go to Alerts > Notification Policies.
- Click the notification policy you want to delete. The notification policy
definition page displays:
- Click Edit Policy.
- Click Delete notification policy.
- Click Delete to confirm that you want to delete the notification policy.
Override notification policy defaults
You can override notification policy defaults based on alert labels by adding an
overrides
collection to its definition. Use the alert_label_matchers
collection
to define the conditions for the override.
You can specify either a monitor label name or a signal label name as the value
of
the alert_label_matchers
collection. If you specify any other label type, the
override isn't processed for the specified monitor, even if the alerting series
includes the specified label.
When an override matches the defined conditions, the notification defined in the
specified notifiers
collection triggers. If an override matches multiple
conditions, only the first match triggers a notification. Any additional matches
don't trigger a notification.
When a monitor has signal_per_series
set to true
(multiple alerts) as the signal
grouping, you can use any label as a notification policy override.
To override a notification policy:
- Go to Alerts > Notification Policies.
- On the Notification Policies page, click Create notification policy.
- Enter a name for the notification policy and select a team to own the policy.
- For both Critical Alert Notifiers and Warning Alert Notifiers:
- Select the time period for resending notifications.
- Choose a notifier to define who to notify and through which endpoint.
- Click + Add Override Notifier to define a notification override:
- Enter a label name and value to match on. To add another label and value to match on, click Add Matcher Label.
- Define the conditions for both the Critical Alert Notifiers and Warning Alert Notifiers sections.
- Click Save.
Choose a notification policy
When a monitor triggers, Chronosphere runs checks to determine which notification policy to apply. First, Chronosphere checks the notification policy type to determine:
-
If the monitor has an explicit notification policy, use that policy.
-
If the associated collection has a default policy, use that policy. The
notification_policy_slug
ornotification_policy_id
attribute on a collection entity declares a default policy. -
If the collection doesn't have a default policy but has an owned policy, use that policy. The
bucket_slug
orbucket_id
attribute on a notification policy entity declares an owned policy.Owned policies are deprecated. Refer to migrate buckets to collections to migrate from buckets to collections.
From there, Chronosphere evaluates the alert severity:
- If the alert severity is warning and matches a warning override, use the override notifier. Otherwise, use the default warning notifier.
- If the alert severity is critical and matches a critical override, use the override notifier. Otherwise, use the default critical.
Chronosphere then delivers the alert using the selected notifier type.