Notifiers

When an alert triggers, Chronosphere Observability Platform sends a notification based on a notifier, which is an endpoint you specify that defines where to deliver active alerts and who to notify. When creating a notification policy, you select a notifier that specifies where to route the alert. A single alert results in one notification, which can be of either warn or critical severity. If there are multiple time series, the alert is de-duplicated and activated only once. However, the alert indicates which time series are violating the condition. Each notification policy triggers the sending of new notifications periodically, with a default period of 60 minutes. Special global notifiers receive and send notifications for every alert generated by Observability Platform. Observability Platform supports the following notifiers:

Discard

Create a notifier to discard all notifications.

Email

Create an email notifier to send monitor alerts to a specific email address.

incident.io

Configure an alert source in incident.io and then create a webhook notifier in Observability Platform.

OpsGenie

Create a notifier in OpsGenie.

PagerDuty

Create a service in PagerDuty and then create the notifier in Observability Platform.

Slack

Generate an incoming webhook in Slack and then create a Slack notifier in Observability Platform.

VictorOps

Create an API key in VictorOps and then create a VictorOps notifier in Observability Platform.

Webhook

Create a webhook notifier with a URL that specifies the endpoint to send HTTP POST requests to.

Prerequisites

Some notifier configurations require specific information, such as a service key, an API key, or support intervention. The required information depends on the type of notifier that you’re creating. Refer to each notifier’s configuration to determine the necessary information.

Users can modify Terraform-managed resources only by using Terraform. Learn more.

View notifiers

You can view available notifiers in Observability Platform, or return a list of notifiers using Chronoctl. You can view notifiers created by Terraform in Observability Platform, but you can’t modify them.

Web
Chronoctl
API

To view a notifier:

In the navigation menu select Alerts > Notifiers.
Search for the notifier you want to view and select it from the list.

The Edit Notifier page displays the notifier definition.

Requires Chronoctl version 0.46.0 or later.

To view available notifiers with Chronoctl, use one of the following commands:

To return a complete list of notifier resource definitions as YAML documents, use this command:
```
chronoctl notifiers list
```
To return specific resources by name or slug, pass a comma-delimited list to the --names or --slugs options respectively. For example:
```
chronoctl notifiers list --names LIST
```
Replace LIST with a list of notifier names, such as "Infra Webhook,Email Notifier". Each notifier has a name field in the resource definition, which you include in the list.
To return a single resource definition by slug, use the notifiers read command:
```
chronoctl notifiers read SLUG
```
Replace SLUG with the notifier’s slug, such as email-notifier.

These commands return a Notifier resource, which has the following format:

api_version: v1/config
kind: Notifier
spec:
  slug: email-notifier
  name: Email Notifier
  created_at: "2022-11-17T22:56:59.000Z"
  updated_at: "2022-11-17T22:56:59.000Z"
  email:
    to: your-address@example.com
    html: <b>Alert firing</b>
    text: Alert firing

To complete this action with the Chronosphere API, use the ListNotifiers endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Create a notifier

Notifiers use a custom definition format in YAML. You can use Observability Platform, Chronoctl, or the Chronosphere Terraform provider to create and manage them. Notifiers support variables and templating. After creating a notifier, you can select it when defining a notification policy. The following steps apply to creating notifiers. See each notifier page for examples of creating that specific notifier type.

Web
Chronoctl
Terraform
API

To create a notifier:

In the navigation menu select Alerts > Notifiers.
Click Create notifier.
Enter a descriptive name for the notifier.
Select the type of notifier you want to create.
Enter the required information for the notifier.
Optional: Select Notify when resolved to send a resolved alert notification.
Click Save.

To create one or more notifiers with Chronoctl, create a YAML file to define your notifiers and apply the configuration to your instance. Use the lowercase name of the service as a YAML key to set the service to use.Refer to each notifier type for a full example of a definition. You can also generate an example that contains multiple notifier types with Chronoctl by using this command:

chronoctl notifiers scaffold

To apply a definition, use this command:

chronoctl apply -f FILE_NAME.yaml

Replace FILE_NAME with the name of your notifier YAML file.

Create an alert notifier with Terraform by using the chronosphere_{channel}_alert_notifier type followed by a name in a resource declaration.See each notifier type for a full definition example.

Add the definition to a Terraform file.
Run this command to create the resource:
```
terraform apply
```

To complete this action with the Chronosphere API, use the CreateNotifier endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

To create global notifiers, contact Chronosphere Support.

Edit a notifier

Select from the following methods to edit a notifier.

Web
Chronoctl
Terraform
API

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To edit a notifier:

In the navigation menu select Alerts > Notifiers.
Search for the notifier you want to view and select it from the list.
In the Edit Notifier page, make changes to the notifier definition.
Click Save.

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To edit a notifier with Chronoctl:

Get the resource definition for the notifier you want to edit:
- To return a complete list of notifier resource definitions as YAML documents, use the notifiers list command:
  chronoctl notifiers list
- To return a single resource definition by slug, use this command:
  chronoctl notifiers read SLUG
  Replace SLUG with the notifier’s slug, such as email-notifier.
Modify the YAML file for the notifier you want to update.
Apply your changes:
```
chronoctl apply -f FILE_NAME.yaml
```
Replace FILE_NAME with the name of your notifier YAML file.

Chronoctl updates the notifier’s properties if it has the same slug.

To edit a notifier using Terraform:

Modify the notifier resource’s existing properties.
Run terraform apply to apply the changes.
```
terraform apply
```

To complete this action with the Chronosphere API, use the UpdateNotifier endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

To edit global notifiers, contact Chronosphere Support.

Delete a notifier

Select from the following methods to delete a notifier.

Web
Chronoctl
Terraform
API

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To delete a notifier:

In the navigation menu select Alerts > Notifiers.
Search for the notifier you want to delete and select it from the list.
In the Edit Notifier page, click the delete icon .
In the confirmation dialog, click Delete.

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To delete a notifier with Chronoctl, run this command:

chronoctl notifiers delete SLUG

Replace SLUG with the slug of the notifier you want to delete.

To delete a resource that’s managed by Terraform:

Edit your Terraform configuration file to remove the pre-existing resource definition.
Run this command to remove the resource from Observability Platform:
```
terraform apply
```

To complete this action with the Chronosphere API, use the DeleteNotifier endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

To delete global notifiers, contact Chronosphere Support.

Notifications for alerts in third-party services

By default, Observability Platform sends a notification when an alert for a monitor resolves in third-party services, such as PagerDuty and VictorOps. You can choose to not send notifications by configuring the specified parameter in the Chronoctl YAML definition or Terraform resource for your notifier.

If you create a notifier with Observability Platform, notifications are never sent when an alert resolves. You can modify this behavior only for notifiers created using Chronoctl, Terraform, or the Chronosphere API.

Chronoctl
Terraform

To skip notifications when alerts for a monitor resolve, set the value of the skip_resolved property to true in your notifier definition.For example, the following definition skips sending notifications for resolved alerts for an Opsgenie notifier:

api_version: v1/config
kind: Notifier
spec:
  name: test-opsgenie
  slug: test-opsgenie
  skip_resolved: true
    opsgenie:
…

To skip notifications when alerts for a monitor resolve, set the value of the send_resolved key to false. If you don’t explicitly set a value for this key, Observability Platform sends notifications for resolved alerts by default. The send_resolved key is a top-level key.For example, the following resource skips sending notifications for resolved alerts for an Opsgenie notifier:

resource "chronosphere_opsgenie_alert_notifier" "opsgenie_notifier" {
  name = "Opsgenie Notifier"
  …
  send_resolved = false
  …
}

Use variables in notifiers

Notification content supports Go templates and Prometheus Alertmanager variables. Notifiers support many features and variables demonstrated in Alertmanager notification examples.

Available variables

You can reference Prometheus Alertmanager variables with the {{.VARIABLE_NAME }} syntax. Variable references work in the notifier’s definition and the notification’s contents. Notifiers can access monitor labels by using variables with the {{ .CommonLabels.LABEL }} pattern, and from the alerting metric with the {{ .Labels.LABEL }} pattern. In both patterns, replace LABEL with the label’s name. Similarly, you can reference monitor-defined annotations by using the {{ .CommonAnnotations.ANNOTATION }} pattern, replacing ANNOTATION with the monitor annotation’s name.

If your monitor query includes a label that generates different annotation values, then annotations won’t be included in CommonAnnotations or CommonLabels.

See the Alertmanager documentation for a reference list of alerting variables and templating functions.

Variable usage examples

This Slack notifier’s channel value takes advantage of monitor-defined labels to post a notification to a channel based on a triggering metric’s label:

Chronoctl
Terraform

api_version: v1/config
kind: Notifier
spec:
  notifier:
    name: slack-notifier
    slug: slack-notifier
    slack:
      …
      channel: "metric-notification-{{ .CommonLabels.<LABEL> }}"
      …

resource "chronosphere_slack_alert_notifier" "slack_notifier" {
  name = "Slack Notifier"
  …
  channel = "metric-notification-{{ .CommonLabels.<LABEL> }}"
  …
}

You can also reference annotations defined in the monitor:

Chronoctl
Terraform

api_version: v1/config
kind: Notifier
spec:
  notifier:
    name: slack-notifier
    slug: slack-notifier
    slack:
      …
      text: "Summary: {{ .CommonAnnotations.summary }}"
      …

resource "chronosphere_slack_alert_notifier" "slack_notifier" {
  name = "Slack Notifier"
  …
  text = "Summary: {{ .CommonAnnotations.summary }}"
  …
}

You can also use multiple labels in notifiers. The following example uses multiple labels for an OpsGenie notifier:

Chronoctl

api_version: v1/config
kind: Notifier
spec:
  ops_genie:
    priority: '{{ .CommonLabels.Priority }}'

kind: Monitor
spec:
  labels:
    Priority: P2

The priority in the outgoing notification is correctly templated with priority P2.

Global notifiers

Instead of using a notification policy to define when a notification is sent, global notifiers receive and send notifications for every alert generated across Observability Platform. Since these notifiers can generate a constant feed of many notifications, they must be created and managed exclusively by Chronosphere Support. You can’t edit or delete global notifiers on your own, and Observability Platform designates this with a banner in notifier editing interfaces. In the Notifiers list, Observability Platform designates global notifiers with a Global chip displayed with their type.

Observability Platform

Telemetry Pipeline

Tools

More information

Discard

Email

incident.io

OpsGenie

PagerDuty

Slack

VictorOps

Webhook

Prerequisites

View notifiers

Create a notifier

Edit a notifier

Delete a notifier

Notifications for alerts in third-party services

Use variables in notifiers

Available variables

Variable usage examples

Global notifiers

Observability Platform

Telemetry Pipeline

Tools

More information

Discard

Email

incident.io

OpsGenie

PagerDuty

Slack

VictorOps

Webhook

​Prerequisites

​View notifiers

​Create a notifier

​Edit a notifier

​Delete a notifier

​Notifications for alerts in third-party services

​Use variables in notifiers

​Available variables

​Variable usage examples

​Global notifiers

Prerequisites

View notifiers

Create a notifier

Edit a notifier

Delete a notifier

Notifications for alerts in third-party services

Use variables in notifiers

Available variables

Variable usage examples

Global notifiers