Create alerts for log queries

💡

If you're looking for documentation about monitor alerts in Observability Platform, see View and create monitors.

Use alerts to notify individuals when a Logs query matches a particular event. You create alerts by defining a query to match on and an action to trigger when LogScale identifies a matching event in the alert query.

Unlike scheduled searches, each alert query runs continuously, which causes an action to trigger immediately when an event matches the query. Alerts can be one of the following types:

Standard alerts trigger when an aggregate query generates a result set, which includes one or more rows of results. For example, the following query matches any logs in the production environment where the http_response tag isn't 200, and groups the results by the http_path and http_response tags:
```
environment = production
#http_response != 200
| groupBy([http_path, http_response])
```
Filter alerts trigger when the alert query filters an event, causing each matching event to trigger an alert. For example, the following query matches any logs where the severity equals ERROR for the payment_service service in the production environment:
```
severity = ERROR
|service="payment_service"
|env="production"
```

For more information about alerts, see Alerts (opens in a new tab) in the LogScale documentation.

Prerequisites

Create an action in LogScale to initiate when an alert triggers, such as notifying a PagerDuty group or sending a message to a Slack channel. You can create the alert first, but then need to create an action and edit the alert to assign an action to it.

Create an alert

Select from the following methods to create an alert for a logs query.

To create an alert in Observability Platform:

In the navigation menu select Explorers > Logs Explorer.
Click Logs Automation to display the Logs alerting capabilities.
On the Logs Automation page, click Alerts and then click New alert to display the new alert pane.
In the query box, enter a query for the alert to match on.
In the Details pane, enter a name for the alert, and include any variables you want to use as placeholders in the alert name. See message template and variables (opens in a new tab) in the LogScale documentation for more information.
Select an action to determine what happens when the alert triggers.
In the Query dropdown, select Legacy or Aggregate as your alert type.
Set a throttle period to control how often an alert triggers. See setting alert throttle period (opens in a new tab) in the LogScale documentation for more information.

Setting a throttle period isn't available for filter alerts.
Click Save to create your alert.

When the alert query matches events, the action associated with the alert triggers and notifies the groups or individuals defined in the action.

Terraform alert examples

The following examples provide Terraform resources for defining a standard alert and a filter alert.

For the run_as_user field, enter the email address of the user to run the query on behalf of. The email address must map to an existing user email address in LogScale. The format of the email address is:

[USERNAME]+[CHRONOSPHERE_TENANT]@[EMAIL_DOMAIN]

For example, a user named cdanvers whose email is cdanvers@example.com in the acme Chronosphere tenant would result in the cdanvers+acme@example.com email address for the run_as_user field.

Standard alert

The following resource definition creates an alert that Terraform refers to as count_errors_alert. This example defines a standard alert (opens in a new tab), which returns results from an aggregate query.

logscale_standard_alert.tf

resource "chronosphere_logscale_alert" "count_errors_alert" {
  # Repository where the alert query runs.
  repository = "my-repository"
  # Display name of the alert
  name = "Alert"
  # Description for the alert.
  description = "Severity alert count"
  # Alert type, which can be STANDARD or FILTER.
  alert_type = "STANDARD"
  # Defined query that generates the alert when conditions are met.
  # The query cannot contain aggregate functions when defining an alert with
  # alert_type = "FILTER".
  query = "severity = ERROR | count(as=numErrors) | numErrors > 500"
  # Time window for how often the query runs. Default: 1h.
  time_window = "1h"
  # How often an alert is set to trigger. Default: 1h.
  throttle_duration = "60m"
  # Optional field to throttle alerts by. When the alert triggers, no
  # further events with the same values for the selected field are sent to # # the associated actions within the throttle period.
  throttle_field = "some_field_to_throttle_by"
  # Optional labels to assign to the alert, which you can use to group alerts by.
  tags = [
    "tag1",
    "tag2",
  ]
  # The email address of the user to run the query on behalf of.
  run_as_user = "cdanvers+acme@example.com"
  disabled = false
  # The IDs of the actions to run when the query for the alert triggers.
  action_ids = [
    chronosphere_logscale_action.example_email_action.id,
    chronosphere_logscale_action.example_slack_action.id,
  ]
}

Filter alert

The following resource definition creates an alert that Terraform refers to as example_filter_alert. This example defines a filter alert (opens in a new tab), which triggers when the corresponding query filters an event. Each matching event triggers the alert.

logscale_filter_alert.tf

resource "chronosphere_logscale_alert" "example_filter_alert" {
  # Repository where the alert query runs.
  repository = "my-repository"
  # Display name of the alert
  name = "Alert"
  # Description for the alert.
  description = "Severity alert"
  # Alert type, which can be STANDARD or FILTER.
  alert_type = "FILTER"
  # Defined query that generates the alert when conditions are met.
  # The query cannot contain aggregate functions when defining an alert with
  # alert_type = "FILTER".
  query = "severity = ERROR"
  # How often an alert is set to trigger. Default: 1h.
  throttle_duration = "60m"
  # Optional field to throttle alerts by. When the alert triggers, no
  # further events with the same values for the selected field are sent to # # the associated actions within the throttle period.
  throttle_field = "some_field_to_throttle_by"
  # Optional labels to assign to the alert, which you can use to group alerts by.
  tags = [
    "tag1",
    "tag2",
  ]
  # The email address of the user to run the query on behalf of.
  run_as_user = "email@example.com"
  disabled = false
  # The IDs of the actions to run when the query for the alert triggers.
  action_ids = [
    chronosphere_logscale_action.example_email_action.id,
    chronosphere_logscale_action.example_slack_action.id,
  ]
}

Actions Scheduled searches