Notifiers

When an alert triggers, it sends a notification based on a notifier, which is an endpoint you specify that defines where to deliver active alerts and who to notify. When creating a notification policy, you select a notifier that specifies where to route the alert.

Notifiers trigger every 60 minutes by default, meaning for every 60 minutes an alert triggers, a new notification sends for each defined notification policy.

A single alert results in one notification, which can be either warn or critical. If there are multiple time series, the alert is de-duplicated, and activates only once. However, the alert indicates which time series are violating the condition.

Each notifier uses a custom definition format in YAML. Use the Chronosphere app, Chronoctl, or Chronosphere Terraform provider to create and manage them.

Notifiers support variables and templating.

Prerequisites

Some notifier configurations require specific information, such as a service key or an API key. The required information depends on the type of notifier that you're creating. Refer to each notifier's configuration to determine the necessary information.

Chronosphere prevents users from modifying Terraform-managed resources in the user interface, with Chronoctl, or by using the API. For details, see the Terraform provider documentation.

Create a notifier

Notifiers use a custom definition format in YAML. You can use the Chronosphere app, Chronoctl, or the Chronosphere Terraform provider to create and manage them.

After creating a notifier, you can select it when defining a notification policy.

Chronosphere supports the following notifiers:

To create a notifier:

  1. In the navigation menu select Alerting > Notifiers.
  2. Click + Create Notifier.
  3. Select the type of notifier that you want to create.
  4. Enter the required information for the notifier.
  5. Click Save.

Email notifier

Enter the following values in the Chronosphere app:

  • Add the destination email address to the Email field.
  • Add an optional HTML or text body to the Body field.

PagerDuty notifier

To configure a PagerDuty notifier:

  1. In PagerDuty, go to Services > Service Directory and click New Service.
  2. Enter a name and description for the service and follow the remaining prompts.
  3. Select Prometheus from the integration types list and create the service.
  4. After PagerDuty redirects you to the new service, click the Integrations tab, which displays the integration key.

Enter the following values in the Chronosphere app:

  • Use the provided value of the key in the Integration Key field.

  • Enter https://events.pagerduty.com/v2/enqueue in the Integration URL field.

    ⚠️

    Don't use the default value in the Integration URL field. You must enter the specified URL or the PagerDuty integration will return an error message.

Slack notifier

To configure a Slack notifier, generate an incoming webhook. Refer to the Slack documentation (opens in a new tab) for instructions.

Enter the following values in the Chronosphere app:

  • Use the value of the key for the API URL field.
  • Add an optional Slack channel to post in to the Channel field.
  • Add an optional Slack user to post in to the Username field.

Webhook notifier

To configure a webhook notifier, enter a URL that specifies the endpoint to send HTTP POST requests to.

  1. Create a notifier.

  2. Select Webhook as the notifier type.

  3. In the URL field, enter a URL, which is called as a POST request. For example, https://webhook.site/3723e8cb-b1b4-4399-86b0-e37f36a2acc5.

Webhook notifier request body

The following is an example body for a POST request to a webhook. It consists of one alert triggered called "test alert" that has several labels associated with it.

Labels that are common across all alerts in a notification display under commonLabels, and then each alerting series includes the full list of labels. Any static labels that you define also display under commonLabels.

Label values for the related signal grouping display under groupLabels, in addition to a "severity" for the label.

The fingerprint is a representation of an alerting series, expressed as a deterministic value based on the hash of the labels.

A single monitor can trigger more than one alert even if you configure multiple alerts with different labels. When sending notifications, the signal groups alerts based on whether the signal grouping is per monitor (one alert) or per signal group (multiple alerts).

{
  "notifier": "test webhook",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "test alert",
        "component": "remote_write",
        "instance": "localhost:3030",
        "job": "collector_binary",
        "severity": "critical",
        "pod_name": "prom-74cbfb46c9-2ftk9"
      },
      "annotations": {
        "ruleid": "32bb3fbe-c10b-44bb-a4c0-3d053f4a08cd",
        "monitor_slug": "test-monitor",
        "notification_policy_slug": "test-policy"
      },
      "startsAt": "2020-05-19T13:57:21.68227886Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "fingerprint": "7424223989b20025"
    }
  ],
  "groupLabels": {
    "alertname": "test alert",
    "severity": "critical"
  },
  "commonLabels": {
    "alertname": "test alert",
    "component": "remote_write",
    "instance": "localhost:3030",
    "job": "collector_binary",
    "severity": "critical",
    "pod_name": "prom-74cbfb46c9-2ftk9"
  },
  "commonAnnotations": {
    "ruleid": "32bb3fbe-c10b-44bb-a4c0-3d053f4a08cd",
    "monitor_slug": "test-monitor",
    "notification_policy_slug": "test-policy"
  },
  "version": "4"
}

Opsgenie notifier

To configure an Opsgenie notifier, generate an API Key. Refer to the Opsgenie documentation (opens in a new tab) for instructions.

Enter the following values in the Chronosphere app:

  • Use the value of the key for the API Key field.
  • Use https://api.opsgenie.com/ for the API URL field.

VictorOps notifier

To configure a VictorOps notifier:

  1. Open the Integrations tab.
  2. Search for "Prometheus" and click Enable Integration.
  3. Copy the Service API Key value.
  4. Open the pre-configured "REST" integration, and copy the URL to notify value.
  5. In the navigation menu select Settings > Routing Keys and review your routing keys. If you have any routing keys defined, copy the appropriate value for the team you want to route this notifier to.

Enter the following values in the Chronosphere app:

  • Use the value of the service API key for the API Key field.
  • Use the URL value for the API URL field.
  • Use the routing key value for the Routing Key field.

Blackhole notifier

The Blackhole notifier routes notifications to nowhere. You can only create this type of notifier in Chronoctl YAML or Terraform.

api_version: v1/config
kind: Notifier
spec:
  name: Example Blackhole
  slug: example-blackhole

Resolve alerts in third-party services

To receive a resolved notification when it resolves in Chronosphere, set send_resolved to true in the base_config key under the service key.

For example, to resolve a monitor with an Opsgenie notifier, create the definition:

api_version: v1/config
kind: Notifier
spec:
  name: test-opsgenie
  slug: test-opsgenie
  skip_resolved: true
  opsgenie:
  
 

Use variables in notifiers

Notification content supports Go templates (opens in a new tab) and Prometheus Alertmanager variables (opens in a new tab). Notifiers support many features and variables demonstrated in Alertmanager notification examples (opens in a new tab).

Available variables

You can reference Alertmanager variables with the syntax {{ .<VARIABLE_NAME> }}. Variable references work in the notifier's definition and the notification's contents.

Notifiers can access monitor labels by using variables with the {{ .CommonLabels.<LABEL> }} pattern, and from the alerting metric with the {{ .Labels.<LABEL> }} pattern. In both patterns, replace <LABEL> with the label's name.

Similarly, you can reference monitor-defined annotations by using the {{ .CommonAnnotations.<ANNOTATION> }} pattern, replacing <ANNOTATION> with the monitor annotation's name.

For a reference of alerting variables and templating functions, refer to the Alertmanager documentation (opens in a new tab).

Variable usage examples

This Slack notifier's channel value takes advantage of monitor-defined labels to post a notification to a channel based on a triggering metric's label:

api_version: v1/config
kind: Notifier
spec:
  notifier:
    name: slack-notifier
    slug: slack-notifier
    slack:
      
      channel: "metric-notification-{{ .CommonLabels.<LABEL> }}"
      

You can also reference annotations defined in the monitor:

api_version: v1/config
kind: Notifier
spec:
  notifier:
    name: slack-notifier
    slug: slack-notifier
    slack:
      
      text: "Summary: {{ .CommonAnnotations.summary }}"