Alerting
Notifications
Notifiers

Notifiers

When an alert triggers, it sends a notification based on a notifier, which is an endpoint you specify that defines where to deliver active alerts and who to notify. When creating a notification policy, you select a notifier that specifies where to route the alert.

Notifiers trigger every 60 minutes by default, meaning for every 60 minutes an alert triggers, a new notification sends for each defined notification policy.

A single alert results in one notification, which can be either warn or critical. If there are multiple time series, the alert is de-duplicated, and activates only once. However, the alert indicates which time series are violating the condition.

Chronosphere supports the following notifiers:

Prerequisites

Some notifier configurations require specific information, such as a service key or an API key. The required information depends on the type of notifier that you're creating. Refer to each notifier's configuration to determine the necessary information.

Users cannot modify Terraform-managed resources in the Chronosphere app, with Chronoctl, or by using the API. Learn more.

View notifiers

You can view available notifiers in the Chronosphere app, or return a list of notifiers using Chronoctl. You can view notifiers created by Terraform in the Chronosphere app, but you can't modify them.

To view a notifier:

  1. In the navigation menu select Alerts > Notifiers.
  2. Search for the notifier you want to view and select it from the list.

The Edit Notifier page displays the notifier definition.

Create a notifier

Notifiers use a custom definition format in YAML. You can use the Chronosphere app, Chronoctl, or the Chronosphere Terraform provider to create and manage them. Notifiers support variables and templating.

After creating a notifier, you can select it when defining a notification policy.

The following steps apply to creating notifiers. See each notifier page for examples of creating that specific notifier type.

To create a notifier:

  1. In the navigation menu select Alerts > Notifiers.
  2. Click Create Notifier.
  3. Enter a descriptive name for the notifier.
  4. Select the type of notifier you want to create.
  5. Enter the required information for the notifier.
  6. Click Save.

Edit a notifier

You can edit a notifier using the Chronosphere app, Chronoctl, or Terraform.

Users cannot modify Terraform-managed resources in the Chronosphere app, with Chronoctl, or by using the API. Learn more.

To edit a notifier:

  1. In the navigation menu select Alerts > Notifiers.
  2. Search for the notifier you want to view and select it from the list.
  3. In the Edit Notifier page, make changes to the notifier definition.
  4. Click Save.

Delete a notifier

You can delete a notifier using the Chronosphere app, Chronoctl, or Terraform.

Users cannot modify Terraform-managed resources in the Chronosphere app, with Chronoctl, or by using the API. Learn more.

To delete a notifier:

  1. In the navigation menu select Alerts > Notifiers.
  2. Search for the notifier you want to delete and select it from the list.
  3. In the Edit Notifier page, click the delete icon .
  4. In the confirmation dialog, click Delete.

Notifications for alerts in third-party services

By default, Chronosphere sends a notification when an alert for a monitor resolves in third-party services, such as PagerDuty and VictorOps. You can choose to not send notifications by configuring the specified parameter in the Chronoctl YAML definition or Terraform resource for your notifier.

If you create a notifier with the Chronosphere app, notifications are never sent when an alert resolves. You can modify this behavior only for notifiers created using Chronoctl, Terraform, or the Chronosphere API.

To skip notifications when alerts for a monitor resolve, set the value of the skip_resolved property to true in your notifier definition.

For example, the following definition skips sending notifications for resolved alerts for an Opsgenie notifier:

api_version: v1/config
kind: Notifier
spec:
  name: test-opsgenie
  slug: test-opsgenie
  skip_resolved: true
    opsgenie:
…

Use variables in notifiers

Notification content supports Go templates (opens in a new tab) and Prometheus Alertmanager variables (opens in a new tab). Notifiers support many features and variables demonstrated in Alertmanager notification examples (opens in a new tab).

Available variables

You can reference Prometheus Alertmanager variables with the {{.VARIABLE_NAME }} syntax. Variable references work in the notifier's definition and the notification's contents.

Notifiers can access monitor labels by using variables with the {{ .CommonLabels.LABEL }} pattern, and from the alerting metric with the {{ .Labels.LABEL }} pattern. In both patterns, replace LABEL with the label's name.

Similarly, you can reference monitor-defined annotations by using the {{ .CommonAnnotations.ANNOTATION }} pattern, replacing ANNOTATION with the monitor annotation's name.

See the Alertmanager documentation (opens in a new tab) for a reference list of alerting variables and templating functions.

Variable usage examples

This Slack notifier's channel value takes advantage of monitor-defined labels to post a notification to a channel based on a triggering metric's label:

api_version: v1/config
kind: Notifier
spec:
  notifier:
    name: slack-notifier
    slug: slack-notifier
    slack:
      …
      channel: "metric-notification-{{ .CommonLabels.<LABEL> }}"
      …

You can also reference annotations defined in the monitor:

api_version: v1/config
kind: Notifier
spec:
  notifier:
    name: slack-notifier
    slug: slack-notifier
    slack:
      …
      text: "Summary: {{ .CommonAnnotations.summary }}"
      …
NotificationsDiscard