A notification policy connects a monitor with notifiers at warning and critical levels. The notification policy applies rules that define how to route an alert when it triggers, such as who to notify and through which notifier type. Chronosphere runs several checks to determine which notification policy to apply. You can define custom overrides for notification policies based on a signal's labels.
To view available notification policies, in Chronoctl
notification-policies list command:
chronoctl notification-policies list
This returns a list of YAML documents, each representing a notification policy. For
example, a Chronoctl listing of a notification policy in the
might look like this:
name: Example Policy
- type: EXACT
- type: REGEX
Chronoctl versions 0.46.0 and later let you apply this YAML output.
Teams create and own notification policies, and each team can own multiple notification policies.
You can set a notification policy in these ways:
- Per collection: If you configure a notification policy as the default policy on a collection, every monitor in that collection inherits the notification policy.
- Per monitor: You can set a notification policy explicitly on each monitor rather than setting a default policy on a collection.
To create a notification policy:
- Go to Alerting > Notification Policies.
- On the Notification Policies page, click + Add.
- Enter a name for the notification policy and select a team to own the policy.
- For both Critical Alert Notifiers and Warning Alert Notifiers:
- Select the time period for resending notifications.
- Choose a notifier to define who to notify and through which endpoint.
- Optional: To add a notifier override, click + Add Override Notifier and specify the key/value pair to match on and the notifier conditions to alert on.
- Click Save.
You can override notification policy defaults based on alert labels by adding an
overrides collection to its definition. Use the
to define the conditions for the override.
You can specify either a monitor label name or a signal group label name as the
value of the
alert_label_matchers collection. If you specify any other label
type, the override isn't processed for the specified monitor, even if the alerting
series includes the specified label.
When an override matches the defined conditions, the notification defined in the
notifiers collection triggers. If an override matches multiple
conditions, only the first match triggers a notification. Any additional matches
don't trigger a notification.
When a monitor has
signal_per_series set to
true (multiple alerts) as the signal
grouping, you can use any label as a notification policy override.
For example, the following YAML definition adds an override notifier when the
component label has the value of
mysql, and another if the
importance label has
This example uses the
EXACT for exact label matching. To use regular
expression matching, provide the
REGEX. These are the resource's
# If an alert has the component=mysql label, route to the db team.
- type: EXACT,
- type: EXACT
When a monitor triggers, Chronosphere runs checks to determine which notification policy to apply. First, Chronosphere checks the notification policy type to determine:
If the monitor has an explicit notification policy, use that policy.
If the associated collection has a default policy, use that policy. The
notification_policy_idattribute on a collection entity declares a default policy.
If the collection doesn't have a default policy but has an owned policy, use that policy. The
bucket_idattribute on a notification policy entity declares an owned policy.
Owned policies are deprecated. Refer to migrate buckets to collections to migrate from buckets to collections.
From there, Chronosphere evaluates the alert severity:
- If the alert severity is warning and matches a warning override, use the override notifier. Otherwise, use the default warning notifier.
- If the alert severity is critical and matches a critical override, use the override notifier. Otherwise, use the default critical.
Chronosphere then delivers the alert using the selected notifier type.