One reason to ingest and store time series data is to know when data does or doesn’t
meet certain criteria. To generate alerts and notifications from data, regardless of
whether the data’s for your services or your use of Chronosphere Observability Platform
itself, use alerting.You create monitors to query time series, and optionally group results
into signals. Whenever a time series meets a condition, an
alert triggers that sends a notification. You can mute
notifications with muting rules.
Observability Platform alerting revolves around monitors. You can
watch time series data that’s specified in a monitor query. The results of a monitor
query can be time series data or the outcome of a function applied to time series
data. Because Observability Platform is a single data store, you can merge alerts,
which lets an alert query all metrics and not only metrics local to a Prometheus
instance.
Conditions are evaluations run against
every time series from the resulting monitor query, along with a related severity,
such as critical or warning. For example, conditions can help you determine if a time
series value remains greater than 1 for more than 30 seconds. If this happens,
Observability Platform can display a warning alert.You can also specify condition overrides
to override the default conditions
that define when an alert triggers for a monitor.
Signals are optional groupings by label of the
results returned from a monitor query. For example, you can use signals to group time
series from multiple environments, such as development or production, to generate
separate alerts for each environment.In Observability Platform, on the Monitors page, monitors with defined signals
display the file tree icon. To view the signals
from a displayed monitor, click the name of the monitor from the list.Observability Platform supports converting from Prometheus alerts to signals, and
strongly recommends using signals instead of configuring complex routing trees to
route notifications:
You can configure signals for individual monitors, which isolates changes to only
that monitor. Prometheus requires a single global configuration, and any change
impacts users across your entire organization.
Signals let you group notifications within the monitor itself, rather than grouping
notifications within the notification routing tree. This model makes signals easier
to manage as part of a monitor configuration.
Alerts occur when one or more monitor query results
trigger a condition. By configuring the monitor signal to group query results, you
can generate an alert for each signal that triggers a condition.
When an alert triggers, and optionally when an alert resolves, Observability Platform
sends a notification to specified endpoints called
notifiers, including PagerDuty, email, and Slack.
You configure your notifications with
notification policies.
Analyze patterns across alerts for
individual monitors, and alerts that span across multiple monitors, to better
understand where issues are occurring in your system.
