Alerts

About monitors and alerts

One reason to ingest and store time series data is to know when data does or doesn't meet certain criteria. To generate alerts and notifications from data, regardless of whether it's about your system or about your use of Chronosphere itself, use alerting.

Chronosphere lets you create monitors to query time series, and optionally group results into signal groups. Whenever a time series meets a condition, an alert triggers that sends a notification. You can mute notifications with muting rules.

Each team can own many collections, which are a group of resources like monitors and dashboards. The identified team owns the systems that the associated dashboards and monitors track. Each collection can have either a default notification policy for all monitors, or you can set notification policies explicitly for each monitor. These policies apply a function to time series data that determines whether an alert triggers.

Notification policy flow

Monitors

Chronosphere alerting is based on monitors. You can watch time series data that's specified in a monitor query. The results of a monitor query can be time series data or the outcome of a function applied to time series data. Because Chronosphere is a single data store, you can merge alerts, which lets an alert query all metrics and not only metrics local to a Prometheus instance.

Signals

Signals are optional groupings by label of the results returned from a monitor query. For example, you can use signals to group time series from multiple environments, such as development or production, to generate separate alerts for each environment.

Chronosphere supports converting from Prometheus alerts to signals, and strongly recommends using signals instead of configuring complex routing trees to route notifications:

  • You can configure signals for individual monitors, which isolates changes to only that monitor. Prometheus requires a single global configuration, and any change impacts users across your entire organization.
  • Signals let you group notifications within the monitor itself, rather than grouping notifications within the notification routing tree. This model makes signals easier to manage as part of a monitor configuration.

Conditions

Conditions are evaluations run against every time series from the resulting monitor query, along with a related severity, such as critical or warning. For example, conditions can help you determine if a time series value remains greater than 1 for more than 30 seconds. If this happens, Chronosphere can display a warning alert.

Alerts

Alerts occur when one or more monitor query results trigger a condition. Configuring the monitor signal to group query results generates an alert for each signal that triggers a condition.

Notifications

When an alert triggers, and optionally when an alert resolves, Chronosphere sends a notification to specified endpoints called notifiers, including PagerDuty, email, and Slack. You configure your notifications with notification policies.

Muting rules

Using Chronosphere, you can temporarily mute notifications for a triggered alert. These muting rules can apply to:

  • A specific alert.
  • The monitor or signal that generated the alert.
  • Any stored time series.