Alerts

Create alerts for log queries

💡

If you're looking for documentation about monitor alerts in Observability Platform, see View and create monitors.

Use alerts to notify individuals when a Logs query matches a particular event. You create alerts by defining a query to match on and an action to trigger when LogScale identifies a matching event in the alert query.

Unlike scheduled searches, each alert query runs continuously, which causes an action to trigger immediately when an event matches the query. Alerts can be one of the following types:

Standard alerts trigger when an aggregate query generates a result set, which includes one or more rows of results. For example, the following query matches any logs in the production environment where the http_response tag isn't 200, and groups the results by the http_path and http_response tags:
```
environment = production
#http_response != 200
| groupBy([http_path, http_response])
```
Filter alerts trigger when the alert query filters an event, causing each matching event to trigger an alert. For example, the following query matches any logs where the severity equals ERROR for the payment_service service in the production environment:
```
severity = ERROR
|service="payment_service"
|env="production"
```

For more information about alerts, see Alerts (opens in a new tab) in the LogScale documentation.

Prerequisites

Create an action in LogScale to initiate when an alert triggers, such as notifying a PagerDuty group or sending a message to a Slack channel. You can create the alert first, but then need to create an action and edit the alert to assign an action to it.

Create an alert

Use one of the following method to create an alert for a logs query.

To create an alert in Observability Platform:

In the navigation menu select Explorers > Logs Explorer.
Click Logs Automation to display the Logs alerting capabilities.
On the Logs Automation page, click Alerts and then click New alert.
In the New alert pane, select Standard alert or Filter alert as your alert type.
Enter a name for the alert, and include any variables you want to use as placeholders in the alert name. See message template and variables (opens in a new tab) in the LogScale documentation for more information.
In the Query section, enter a query for the alert to match on.
Select an action to determine what happens when the alert triggers.
Set a throttle period to control how often an alert triggers. See setting alert throttle period (opens in a new tab) in the LogScale documentation for more information.

Setting a throttle period isn't available for filter alerts.
Click Create alert to create your alert.

When the alert query matches events, the action associated with the alert triggers and notifies the groups or individuals defined in the action.

Terraform alert examples

The following examples provide Terraform resources for defining a standard alert and a filter alert.

Standard alert

The following resource definition creates an alert that Terraform refers to as count_errors_alert. This example defines a standard alert (opens in a new tab), which returns results from an aggregate query.

logscale_standard_alert.tf

resource "chronosphere_logscale_alert" "count_errors_alert" {
  # Repository where the alert query runs.
  repository = "my-repository"
  # Display name of the alert
  name = "Alert"
  # Description for the alert.
  description = "Severity alert count"
  # Alert type, which can be STANDARD or FILTER.
  alert_type = "STANDARD"
  # Defined query that generates the alert when conditions are met.
  # The query cannot contain aggregate functions when defining an alert with
  # alert_type = "FILTER".
  query = "severity = ERROR | count(as=numErrors) | numErrors > 500"
  # Time window for how often the query runs. Default: 1h.
  time_window = "1h"
  # How often an alert is set to trigger. Default: 1h.
  throttle_duration = "60m"
  # Optional field to throttle alerts by. When the alert triggers, no
  # further events with the same values for the selected field are sent to # # the associated actions within the throttle period.
  throttle_field = "some_field_to_throttle_by"
  # Optional labels to assign to the alert, which you can use to group alerts by.
  tags = [
    "tag1",
    "tag2",
  ]
  # The email address of the user to run the query on behalf of.
  run_as_user = "email@example.com"
  disabled = false
  # The IDs of the actions to run when the query for the alert triggers.
  action_ids = [
    chronosphere_logscale_action.example_email_action.id,
    chronosphere_logscale_action.example_slack_action.id,
  ]
}

Filter alert

The following resource definition creates an alert that Terraform refers to as example_filter_alert. This example defines a filter alert (opens in a new tab), which triggers when the corresponding query filters an event. Each matching event triggers the alert.

logscale_filter_alert.tf

resource "chronosphere_logscale_alert" "example_filter_alert" {
  # Repository where the alert query runs.
  repository = "my-repository"
  # Display name of the alert
  name = "Alert"
  # Description for the alert.
  description = "Severity alert"
  # Alert type, which can be STANDARD or FILTER.
  alert_type = "FILTER"
  # Defined query that generates the alert when conditions are met.
  # The query cannot contain aggregate functions when defining an alert with
  # alert_type = "FILTER".
  query = "severity = ERROR"
  # How often an alert is set to trigger. Default: 1h.
  throttle_duration = "60m"
  # Optional field to throttle alerts by. When the alert triggers, no
  # further events with the same values for the selected field are sent to # # the associated actions within the throttle period.
  throttle_field = "some_field_to_throttle_by"
  # Optional labels to assign to the alert, which you can use to group alerts by.
  tags = [
    "tag1",
    "tag2",
  ]
  # The email address of the user to run the query on behalf of.
  run_as_user = "email@example.com"
  disabled = false
  # The IDs of the actions to run when the query for the alert triggers.
  action_ids = [
    chronosphere_logscale_action.example_email_action.id,
    chronosphere_logscale_action.example_slack_action.id,
  ]
}

Actions Scheduled searches