> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Notifiers

When an [alert](/investigate/alerts) triggers, Chronosphere Observability Platform
sends a notification based on a *notifier*, which is an endpoint you specify that
defines where to deliver active alerts and who to notify. When creating a
[notification policy](/investigate/alerts/notifications/policies), you select a
notifier that specifies where to route the alert.

A single alert results in one notification, which can be of either `warn` or `critical`
severity. If there are multiple time series, the alert is de-duplicated and activated
only once. However, the alert indicates which time series are violating the condition.

Each notification policy triggers the sending of new notifications periodically,
with a default period of 60 minutes. Special [global notifiers](#global-notifiers)
receive and send notifications for every alert generated by Observability Platform.

For Slack, PagerDuty, webhooks, VictorOps, or OpsGenie, the
[Chronosphere API](/tooling/api-info) uses inline `destinations` that reference
[external connections](/investigate/alerts/notifications/external-connections),
or legacy `notifier_slugs` as in [Create a notifier](#create-a-notifier) and
[View notifiers](#view-notifiers). You can't set both on the same notifier list.
For rules, migration steps, and field-level destination fields, see
[Notifier resource migration](/investigate/alerts/notifications/external-connections#notifier-resource-migration)
and
[External connections and route destinations](/investigate/alerts/notifications/policies#external-connections-and-route-destinations).

Observability Platform supports the following notifiers:

<Columns cols={2} rowGroup="2">
  <Card horizontal icon="trash" title="Discard" href="/investigate/alerts/notifications/notifiers/discard">
    <p>Create a notifier to discard all notifications.</p>
  </Card>

  <Card horizontal icon="mail" title="Email" href="/investigate/alerts/notifications/notifiers/email">
    <p>Create an email notifier to send monitor alerts to a specific email address.</p>
  </Card>

  <Card horizontal icon="flame" title="incident.io" href="/investigate/alerts/notifications/notifiers/incident">
    <p>Configure an alert source in incident.io and then create a webhook notifier in Observability Platform.</p>
  </Card>

  <Card horizontal icon="atlassian" title="OpsGenie" href="/investigate/alerts/notifications/notifiers/opsgenie">
    <p>Create a notifier in OpsGenie.</p>
  </Card>

  <Card horizontal icon="circle-parking" title="PagerDuty" href="/investigate/alerts/notifications/notifiers/pagerduty">
    <p>Create a service in PagerDuty and then create the notifier in Observability Platform.</p>
  </Card>

  <Card horizontal icon="slack" title="Slack" href="/investigate/alerts/notifications/notifiers/slack">
    <p>Generate an incoming webhook in Slack and then create a Slack notifier in  Observability Platform.</p>
  </Card>

  <Card horizontal icon="leaf" title="VictorOps" href="/investigate/alerts/notifications/notifiers/victorops">
    <p>Create an API key in VictorOps and then create a VictorOps notifier in Observability Platform.</p>
  </Card>

  <Card horizontal icon="webhook" title="Webhook" href="/investigate/alerts/notifications/notifiers/webhook">
    <p>Create a webhook notifier with a URL that specifies the endpoint to send HTTP POST requests to.</p>
  </Card>
</Columns>

## Prerequisites

Some notifier configurations require specific information, such as a service key,
an API key, or support intervention. The required information depends on the type
of notifier that you're creating. Refer to each notifier's configuration to determine
the necessary information.

<Note>
  Users can modify Terraform-managed resources only by using Terraform.
  [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources).
</Note>

## View notifiers

You can view available notifiers in Observability Platform, or return a list of
notifiers using Chronoctl. You can view notifiers created by Terraform in
Observability Platform, but you can't modify them.

<Tabs>
  <Tab title="Web" id="view-a-notifier-web">
    To view a notifier:

    1. In the navigation menu, select
       **<Icon icon="bell" /> Alerting <span aria-label="and then">></span> Notifiers**.
    2. Search for the notifier you want to view and select it from the list.

    The **Edit Notifier** page displays the notifier definition.
  </Tab>

  <Tab title="Chronoctl" id="view-a-notifier-chronoctl">
    To view available notifiers with Chronoctl, use one of the following commands:

    * To return a complete list of notifier resource definitions as YAML documents, use
      this command:

      ```shell theme={null}
      chronoctl notifiers list
      ```

    * To return specific resources by name or slug, pass a
      comma-delimited list to the `--names` or `--slugs` options respectively. For
      example:

      ```shell /LIST/ theme={null}
      chronoctl notifiers list --names LIST
      ```

      Replace *`LIST`* with a list of notifier names, such as `"Infra Webhook,Email Notifier"`.
      Each notifier has a `name` field in the resource definition, which you include in
      the list.

    * To return a single resource definition by slug, use the `notifiers read` command:

      ```shell /SLUG/ theme={null}
      chronoctl notifiers read SLUG
      ```

      Replace *`SLUG`* with the notifier's slug, such as `email-notifier`.

    These commands return a `Notifier` resource, which has the following format:

    ```yaml theme={null}
    api_version: v1/config
    kind: Notifier
    spec:
      slug: email-notifier
      name: Email Notifier
      created_at: "2022-11-17T22:56:59.000Z"
      updated_at: "2022-11-17T22:56:59.000Z"
      email:
        to: your-address@example.com
        html: <b>Alert triggering</b>
        text: Alert triggering
    ```
  </Tab>

  <Tab title="API" id="view-a-notifier-API">
    To complete this action with the Chronosphere API, use the
    [`ListNotifiers`](/tooling/api-info/definition/operations/ListNotifiers) endpoint.

    Because the Chronosphere API requires authentication, include an API token with your
    `curl` request, as shown in the following example. For more details, see
    [Create an API token](/tooling/api-info#create-an-api-token).

    ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
    export CHRONOSPHERE_API_TOKEN="TOKEN"
    export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

    curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
         -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
    ```

    Replace the following:

    * *`TOKEN`*: Your API token.
    * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
    * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
    * *`ENDPOINT_PATH`*: The specific endpoint you want to access.
  </Tab>
</Tabs>

## Create a notifier

Notifiers use a custom definition format in YAML. You can use Observability Platform,
[Chronoctl](/tooling/chronoctl), or the
[Chronosphere Terraform provider](/tooling/infrastructure/terraform) to create and manage them.
Notifiers support [variables](#use-variables-in-notifiers) and templating.

After creating a notifier, you can select it when defining a
[notification policy](/investigate/alerts/notifications/policies).

The following steps apply to creating notifiers. See each notifier page for examples
of creating that specific notifier type.

<Tabs>
  <Tab title="Web" id="create-a-notifier-web">
    To create a notifier:

    1. In the navigation menu, select
       **<Icon icon="bell" /> Alerting <span aria-label="and then">></span> Notifiers**.
    2. Click **Create notifier**.
    3. Enter a descriptive name for the notifier.
    4. Select the type of notifier you want to create.
    5. Enter the required information for the notifier.
    6. Optional: Select **Notify when resolved** to send a resolved alert notification.
    7. Click **Save**.
  </Tab>

  <Tab title="Chronoctl" id="create-a-notifier-chronoctl">
    To create one or more notifiers with [Chronoctl](/tooling/chronoctl), create a YAML
    file to define your notifiers and apply the configuration to your instance. Use the
    lowercase name of the service as a YAML key to set the service to use.

    Refer to each notifier type for a full example of a definition. You can also generate
    an example that contains multiple notifier types with Chronoctl by using this
    command:

    ```shell theme={null}
    chronoctl notifiers scaffold
    ```

    To apply a definition, use this command:

    ```shell /FILE_NAME/ theme={null}
    chronoctl apply -f FILE_NAME.yaml
    ```

    Replace *`FILE_NAME`* with the name of your notifier YAML file.
  </Tab>

  <Tab title="Terraform" id="create-a-notifier-terraform">
    Create an alert notifier with Terraform by using the
    `chronosphere_{channel}_alert_notifier` type followed by a name in a resource
    declaration.

    See each notifier type for a full definition example.

    1. Add the definition to a Terraform file.
    2. Run this command to create the resource:

       ```shell theme={null}
       terraform apply
       ```
  </Tab>

  <Tab title="API" id="create-a-notifier-API">
    To complete this action with the Chronosphere API, use the
    [`CreateNotifier`](/tooling/api-info/definition/operations/CreateNotifier) endpoint.

    Because the Chronosphere API requires authentication, include an API token with your
    `curl` request, as shown in the following example. For more details, see
    [Create an API token](/tooling/api-info#create-an-api-token).

    ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
    export CHRONOSPHERE_API_TOKEN="TOKEN"
    export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

    curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
         -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
    ```

    Replace the following:

    * *`TOKEN`*: Your API token.
    * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
    * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
    * *`ENDPOINT_PATH`*: The specific endpoint you want to access.
  </Tab>
</Tabs>

<Note>
  To create [global notifiers](#global-notifiers), contact [Chronosphere Support](/support).
</Note>

## Edit a notifier

Select from the following methods to edit a notifier.

<Tabs>
  <Tab title="Web" id="edit-a-notifier-web">
    <Note>
      Users can modify Terraform-managed resources only by using Terraform.
      [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources).
    </Note>

    To edit a notifier:

    1. In the navigation menu, select
       **<Icon icon="bell" /> Alerting <span aria-label="and then">></span> Notifiers**.
    2. Search for the notifier you want to view and select it from the list.
    3. In the **Edit Notifier** page, make changes to the notifier definition.
    4. Click **Save**.
  </Tab>

  <Tab title="Chronoctl" id="edit-a-notifier-chronoctl">
    <Note>
      Users can modify Terraform-managed resources only by using Terraform.
      [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources).
    </Note>

    To edit a notifier with [Chronoctl](/tooling/chronoctl):

    1. Get the resource definition for the notifier you want to edit:

       * To return a complete list of notifier resource definitions as YAML documents, use
         the `notifiers list` command:

         ```shell theme={null}
         chronoctl notifiers list
         ```

       * To return a single resource definition by slug, use this command:

         ```shell /SLUG/ theme={null}
         chronoctl notifiers read SLUG
         ```

         Replace *`SLUG`* with the notifier's slug, such as `email-notifier`.

    2. Modify the YAML file for the notifier you want to update.

    3. Apply your changes:

       ```shell /FILE_NAME/ theme={null}
       chronoctl apply -f FILE_NAME.yaml
       ```

       Replace *`FILE_NAME`* with the name of your notifier YAML file.

    Chronoctl updates the notifier's properties if it has the same slug.
  </Tab>

  <Tab title="Terraform" id="edit-a-notifier-terraform">
    To edit a notifier using [Terraform](/tooling/infrastructure/terraform):

    1. Modify the notifier resource's existing properties.
    2. Run `terraform apply` to apply the changes.

       ```shell theme={null}
       terraform apply
       ```
  </Tab>

  <Tab title="API" id="edit-a-notifier-API">
    To complete this action with the Chronosphere API, use the
    [`UpdateNotifier`](/tooling/api-info/definition/operations/UpdateNotifier) endpoint.

    Because the Chronosphere API requires authentication, include an API token with your
    `curl` request, as shown in the following example. For more details, see
    [Create an API token](/tooling/api-info#create-an-api-token).

    ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
    export CHRONOSPHERE_API_TOKEN="TOKEN"
    export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

    curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
         -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
    ```

    Replace the following:

    * *`TOKEN`*: Your API token.
    * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
    * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
    * *`ENDPOINT_PATH`*: The specific endpoint you want to access.
  </Tab>
</Tabs>

<Note>
  To edit [global notifiers](#global-notifiers), contact [Chronosphere Support](/support).
</Note>

## Delete a notifier

Select from the following methods to delete a notifier.

<Tabs>
  <Tab title="Web" id="delete-a-notifier-web">
    <Note>
      Users can modify Terraform-managed resources only by using Terraform.
      [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources).
    </Note>

    To delete a notifier:

    1. In the navigation menu, select
       **<Icon icon="bell" /> Alerting <span aria-label="and then">></span> Notifiers**.
    2. Search for the notifier you want to delete and select it from the list.
    3. In the **Edit Notifier** page, click the delete icon **<Icon icon="trash" />**.
    4. In the confirmation dialog, click **Delete**.
  </Tab>

  <Tab title="Chronoctl" id="delete-a-notifier-chronoctl">
    <Note>
      Users can modify Terraform-managed resources only by using Terraform.
      [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources).
    </Note>

    To delete a notifier with [Chronoctl](/tooling/chronoctl), run this command:

    ```shell /SLUG/ theme={null}
    chronoctl notifiers delete SLUG
    ```

    Replace *`SLUG`* with the slug of the notifier you want to delete.
  </Tab>

  <Tab title="Terraform" id="delete-a-notifier-terraform">
    To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform):

    1. Edit your Terraform configuration file to remove the pre-existing resource
       definition.
    2. Run this command to remove the resource from Observability Platform:

       ```shell theme={null}
       terraform apply
       ```
  </Tab>

  <Tab title="API" id="delete-a-notifier-API">
    To complete this action with the Chronosphere API, use the
    [`DeleteNotifier`](/tooling/api-info/definition/operations/DeleteNotifier) endpoint.

    Because the Chronosphere API requires authentication, include an API token with your
    `curl` request, as shown in the following example. For more details, see
    [Create an API token](/tooling/api-info#create-an-api-token).

    ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
    export CHRONOSPHERE_API_TOKEN="TOKEN"
    export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

    curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
         -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
    ```

    Replace the following:

    * *`TOKEN`*: Your API token.
    * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
    * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
    * *`ENDPOINT_PATH`*: The specific endpoint you want to access.
  </Tab>
</Tabs>

<Note>
  To delete [global notifiers](#global-notifiers), contact [Chronosphere Support](/support).
</Note>

## Notifications for alerts in third-party services

By default, Observability Platform sends a notification when an alert for a monitor
resolves in third-party services, such as PagerDuty and VictorOps. You can choose to
not send notifications by configuring the specified parameter in the Chronoctl YAML
definition or Terraform resource for your notifier.

<Note>
  By default, Observability Platform doesn't send notifications when an alert resolves.
  To send resolved notifications, select **Notify when resolved** when creating a notifier.
  For notifiers created using Chronoctl, Terraform, or the Chronosphere API, use the
  `skip_resolved` or `send_resolved` parameter.
</Note>

<Tabs>
  <Tab title="Chronoctl" id="resolve-alerts-in-third-party-services-chronoctl">
    To skip notifications when alerts for a monitor resolve, set the value of the
    `skip_resolved` property to `true` in your notifier definition.

    For example, the following definition skips sending notifications for resolved alerts
    for an Opsgenie notifier:

    ```yaml theme={null}
    api_version: v1/config
    kind: Notifier
    spec:
      name: test-opsgenie
      slug: test-opsgenie
      skip_resolved: true
        opsgenie:
    ...

    ```
  </Tab>

  <Tab title="Terraform" id="resolve-alerts-in-third-party-services-terraform">
    To skip notifications when alerts for a monitor resolve, set the value of the
    `send_resolved` key to `false`. If you don't explicitly set a value for this key,
    Observability Platform sends notifications for resolved alerts by default. The
    `send_resolved` key is a top-level key.

    For example, the following resource skips sending notifications for resolved alerts
    for an Opsgenie notifier:

    ```terraform theme={null}
    resource "chronosphere_opsgenie_alert_notifier" "opsgenie_notifier" {
      name = "Opsgenie Notifier"
      ...
      send_resolved = false
      ...
    }
    ```
  </Tab>
</Tabs>

## Use variables in notifiers

Notification content supports [Go templates](https://pkg.go.dev/text/template@go1.19) and
[Prometheus Alertmanager variables](https://prometheus.io/docs/alerting/latest/notifications/).
Notifiers support many features and variables demonstrated in
[Alertmanager notification examples](https://prometheus.io/docs/alerting/latest/notification_examples/).

### Available variables

You can reference Prometheus Alertmanager variables with the `{{.VARIABLE_NAME }}`
syntax. Variable references work in the notifier's definition and the notification's
contents.

Notifiers can access monitor labels by using variables with the
`{{ .CommonLabels.LABEL }}` pattern, and from the alerting metric with the
`{{ .Labels.LABEL }}` pattern. In both patterns, replace *`LABEL`* with the
label's name.

Similarly, you can reference monitor-defined annotations by using the
`{{ .CommonAnnotations.ANNOTATION }}` pattern, replacing *`ANNOTATION`* with
the monitor annotation's name.

<Note>
  If your monitor query includes a label that generates different annotation values,
  then annotations won't be included in `CommonAnnotations` or `CommonLabels`.
</Note>

See the
[Alertmanager documentation](https://prometheus.io/docs/alerting/latest/notifications/)
for a reference list of alerting variables and templating functions.

For a simpler way to customize the title and description of notifications
using monitor-specific variables, see
[Notification templates](/investigate/alerts/monitors/notification-templates).

### Variable usage examples

This Slack notifier's `channel` value takes advantage of monitor-defined labels to
post a notification to a channel based on a triggering metric's label:

<Tabs>
  <Tab title="Chronoctl" id="variable-usage-examples-1-chronoctl">
    ```yaml theme={null}
    api_version: v1/config
    kind: Notifier
    spec:
      notifier:
        name: slack-notifier
        slug: slack-notifier
        slack:
          ...
          channel: "metric-notification-{{ .CommonLabels.<LABEL> }}"
          ...
    ```
  </Tab>

  <Tab title="Terraform" id="variable-usage-examples-1-terraform">
    ```terraform theme={null}
    resource "chronosphere_slack_alert_notifier" "slack_notifier" {
      name = "Slack Notifier"
      ...
      channel = "metric-notification-{{ .CommonLabels.<LABEL> }}"
      ...
    }
    ```
  </Tab>
</Tabs>

You can also reference annotations defined in the monitor:

<Tabs>
  <Tab title="Chronoctl" id="variable-usage-examples-2-chronoctl">
    ```yaml theme={null}
    api_version: v1/config
    kind: Notifier
    spec:
      notifier:
        name: slack-notifier
        slug: slack-notifier
        slack:
          ...
          text: "Summary: {{ .CommonAnnotations.summary }}"
          ...
    ```
  </Tab>

  <Tab title="Terraform" id="variable-usage-examples-2-terraform">
    ```terraform theme={null}
    resource "chronosphere_slack_alert_notifier" "slack_notifier" {
      name = "Slack Notifier"
      ...
      text = "Summary: {{ .CommonAnnotations.summary }}"
      ...
    }
    ```
  </Tab>
</Tabs>

You can also use multiple labels in notifiers. The following example uses multiple
labels for an OpsGenie notifier:

<Tabs>
  <Tab title="Chronoctl" id="variable-usage-examples-3-chronoctl">
    ```yaml theme={null}
    api_version: v1/config
    kind: Notifier
    spec:
      ops_genie:
        priority: '{{ .CommonLabels.Priority }}'

    kind: Monitor
    spec:
      labels:
        Priority: P2
    ```

    The priority in the outgoing notification is correctly templated with priority `P2`.
  </Tab>
</Tabs>

## Global notifiers

Instead of using a notification policy to define when a notification is sent, global notifiers
receive and send notifications for every alert generated across Observability Platform.

Since these notifiers can generate a constant feed of many notifications, they must
be created and managed exclusively by [Chronosphere Support](/support). You can't
edit or delete global notifiers on your own, and Observability Platform designates
this with a banner in notifier editing interfaces.

In the Notifiers list, Observability Platform designates global notifiers with a
**Global** chip displayed with their type.