Notification policies
A notification policy connects a monitor with notifiers at warning and critical levels. The notification policy applies rules that define how to route an alert when it triggers, such as who to notify and through which notifier type. Chronosphere Observability Platform runs several checks to determine which notification policy to apply. You can define custom overrides for notification policies based on a signal's labels.
View notification policies
Select from the following methods to view your available notification policies.
To view all notification policies:
- In the navigation menu select Alerts > Notification Policies.
- Use the search box to locate specific notification policies, or choose a notification policy from the Select a Team dropdown.
- Click a notification policy to view its definition.
The list includes notification policies created in Terraform, but you can't modify them.
Create a notification policy
Teams create and own notification policies, and each team can own multiple notification policies.
You can set a notification policy in these ways:
- Per collection: If you configure a notification policy as the default policy on a collection, every monitor in that collection inherits the notification policy.
- Per monitor: You can set a notification policy explicitly on each monitor rather than setting a default policy on a collection.
The repeat interval is the frequency to resend alerts, which is 3600
seconds by
default. If the repeat interval is not set, or is set to 0
, the value defaults to
3600
.
The maximum repeat interval is 40 days. The repeat interval can not be lower than
300
seconds (five minutes), and should be a multiple of 300
. Repeat intervals are
rounded up to the next five minute increment.
You can configure a group_by
field in notification policies both on the default and
override levels. The value of group_by
accepts a set of unique labels. For example,
you can group by env
for environments or by endpoint
.
Grouping notifications with the group_by
field on a notification policy is applied
only if the associated monitor doesn't have any custom
signals configured.
To create a notification policy:
- In the navigation menu select Alerts > Notification Policies.
- On the Notification Policies page, click Create notification policy.
- Enter a name for the notification policy and select a team to own the policy.
- For both Critical Alert Notifiers and Warning Alert Notifiers:
- Select the time period for resending notifications, which defaults to one hour.
- Choose a notifier to define who to notify and through which endpoint.
- Click Save.
Edit a notification policy
Select from the following methods to edit notification policies.
You can use Observability Platform to edit only notification policies created in Observability Platform.
To edit a notification policy:
- In the navigation menu select Alerts > Notification Policies.
- Click the notification policy you want to edit.
- In the notification policy definition page, click Edit Policy.
- Make changes to your notification policy.
- Click Save.
Delete a notification policy
Select from the following methods to delete notification policies.
You can use Observability Platform to delete only notification policies created in Observability Platform.
To delete a notification policy:
- In the navigation menu select Alerts > Notification Policies.
- Click the notification policy you want to delete. The notification policy
definition page displays:
- Click Edit Policy.
- Click Delete notification policy.
- Click Delete to confirm that you want to delete the notification policy.
Override notification policy defaults
You can override notification policy defaults based on alert labels by adding an
overrides
collection to its definition. Use the alert_label_matchers
collection
to define the conditions for the override.
You can specify either a monitor label name or a signal label name as the value
of
the alert_label_matchers
collection. If you specify any other label type, the
override isn't processed for the specified monitor, even if the alerting series
includes the specified label.
When an override matches the defined conditions, the notification defined in the
specified notifiers
collection triggers. If an override matches multiple
conditions, only the first match triggers a notification. Any additional matches
don't trigger a notification.
When a monitor has signal_per_series
set to true
(multiple alerts) as the signal
grouping, you can use any label as a notification policy override.
To override a notification policy:
- In the navigation menu select Alerts > Notification Policies.
- On the Notification Policies page, click Create notification policy.
- Enter a name for the notification policy and select a team to own the policy.
- For both Critical Alert Notifiers and Warning Alert Notifiers:
- Select the time period for resending notifications.
- Choose a notifier to define who to notify and through which endpoint.
- Click + Add Override Notifier to define a notification override:
- Enter a label name and value to match on. To add another label and value to match on, click Add Matcher Label.
- Define the conditions for both the Critical Alert Notifiers and Warning Alert Notifiers sections.
- Click Save.
Choose a notification policy
When a monitor triggers, Observability Platform runs checks to determine which notification policy to apply. First, Observability Platform checks the notification policy type to determine:
-
If the monitor has an explicit notification policy, use that policy.
-
If the associated collection has a default policy, use that policy. The
notification_policy_slug
ornotification_policy_id
attribute on a collection entity declares a default policy. -
If the collection doesn't have a default policy but has an owned policy, use that policy. The
bucket_slug
attribute on a notification policy entity declares an owned policy.Owned policies are deprecated. Refer to migrate buckets to collections to migrate from buckets to collections.
From there, Observability Platform evaluates the alert severity:
- If the alert severity is warning and matches a warning override, use the override notifier. Otherwise, use the default warning notifier.
- If the alert severity is critical and matches a critical override, use the override notifier. Otherwise, use the default critical.
Observability Platform then delivers the alert using the selected notifier type.