Chronosphere Observability Platform lists all triggered alerts and their status, and
generates detailed visualizations of each alert. The visualizations and related tools
help you analyze when, why, and how an alert was triggered.Each alert instance is identified by the unique combination of series labels and
static entity labels that remain after any signal grouping is applied. This means
two triggered alerts are distinct instances when their label sets differ, even if
they come from the same monitor.Each alert instance has a unique URL.Click Copy URL in the page header to copy a shareable URL
to your clipboard. Share it with teammates so they can view the same page directly.The alert details page is mobile-responsive so you can view alert status,
examine chart data, and mute alerts from a mobile device.
To view the list of triggered alerts, select
Alerting > Alerts
in the navigation menu.Each listed alert includes several columns of information:
The date and time that it was Created at
Its Status, designated with an icon, text, and color (see Alert status)
Its title (Alert), as a link to its alert details page
The duration that the alert has been Alerting for
The Configuration entity related to the alert, as a link to the associated
entity, such as a monitor or SLO
You can define the number of Rows per page to display by clicking its dropdown
and selecting a value. To navigate between pages, click the < Go to previous page
and > Go to next page buttons.
Access an alert’s detail page from the alerts list or wherever a triggered
alert is referenced, including from the source entity, such as the
monitor or
service level objective (SLO), that defined the alert’s triggers.
Each alert presents alert details in a dedicated section. This section
includes the alert’s status, a visualization of the triggering queries and
change events, and a table of the series and conditions that triggered the alert.
For status icon and label descriptions, see Alert status.The page also lists any triggering Signal as chips
that identify its keys and their values.
The alert details section provides two tabs for viewing query results:
Stored data: Default. Visualizes the alert’s original query of currently stored
metric data. Results typically match what the alerting engine evaluated, but can
differ if late-arriving data has since been ingested. To account for ingestion
delays, consider
adding an offset to your query.
Evaluated data: Queries the ALERTS_VALUE metric, which records the values the
alerting engine computed at each evaluation interval. See
Alert metrics for reference
documentation about the ALERTS metric and its labels. This view shows what the
monitor actually saw when it made its alerting decision, regardless of data that
arrived later. The table for this tab includes alertstate and severity
columns. A series with alertstate set to pending triggers an alert after it has
been continuously breaching the threshold for the sustain duration configured on
the monitor.
Use the Evaluated data tab to determine if the alerting engine observed data
differently than what’s currently stored, such as when late-arriving metrics change
the stored view.Both tabs display query results as a
time series chart. You can optionally:
Toggle Show thresholds, which draws dotted horizontal lines on the time series chart
depicting the alert’s triggering thresholds.
Toggle Show query or Show queries, which displays the active tab’s underlying queries.
Use the three vertical dots icon to perform additional actions,
including:
Open in explorer, which opens Metrics Explorer
and populates it with the active tab’s query.
Hold the pointer over the chart to display an additional
three vertical dots icon with additional
common panel actions.
The Open in explorer and Show query controls reflect whichever tab is active.
When viewing the Evaluated data tab, the explorer link and query preview display
the ALERTS_VALUE query. When viewing the Stored data tab, they display the
alert’s source query.
In the time series table, select from visualized series to highlight them in the
visualization, and you can use the Search series query box to filter the list.
You can also toggle between two views. The default
table view, lists the time series’ status as an icon and
its labels and their values. The list view lists only each
series’ status and the query that produces it.You can optionally reveal the Conditions that triggered the alert. Observability
Platform lists these conditions as a table of each condition’s status, operator,
sustained duration, time to resolution, and the signals to which the alert’s
conditions apply.
If the alert’s source is an SLO, the alert details page also reproduces the SLO’s
SLI breakdown and Burn/Error rates sections for reference. For more information,
see Service level objectives.
To mute an alert, create a muting rule. You
can do this directly from an alert details page by clicking the Mute alert button.
This opens a panel to create a muting rule
that’s populated with the alert’s relevant source and name, and without requiring
you to leave the alert details page.If a muting rule is already muting a triggered alert, clicking this button instead
opens a panel to edit the associated muting rule.Alternatively, you can select
Alerting > Muting Rules
in the navigation menu to create new muting rules or edit existing muting rules.
However, doing so doesn’t populate the muting rule with details from a specific
alert’s source.
You can customize the chronological scope of an alert’s details by selecting a
time range, which defaults to the Last 1 hour. The chart
and time series table update to depict time series data only within the selected time
range, but the alert’s status always displays its current state.
Selecting a time span that lacks data related to the alert can cause the alert’s
tables or visualizations to report No data. Confirm that the time range selector
is set to a range relevant to the alerting event.
The Alert history section on the alert details page lists past instances of
the same alert. Past instances are previous occurrences with the same signal labels from the same
monitor or SLO.Each past instance appears as a link labeled with the relative time it was created,
such as 2 hours ago. Click any link to navigate to that instance’s alert details
page. The current instance is labeled with the relative time followed by
(This instance) and can be expanded to show the events associated with it.If more recent instances exist outside the initially loaded window, a
Show more recent alerts button appears at the top of the list.
Each alert detail page includes a link to return to the source of its trigger.
For instance, an alert triggered by a monitor has a Return to source monitor link,
and an alert triggered by an SLO has a Return to source SLO link.If you navigate to an alert’s source, you can return to the alert either by clicking
the triggered alert in the source entity. You can also return to the
alerts list and filter it by the alert’s title.
If you’ve enabled change events, the alert details page
displays associated events in the time series chart and lists them in the Change events
section.To configure which change events appear in the chart, click Events
to open the Display events panel. Under Change events:
Toggle Show event markers to show or hide event markers on the chart.
Use the Code or Builder mode toggle to configure the change events query.
In Builder mode, use the category table to select which event categories to display.
Click Save to apply your changes.The change events list includes each Alert change event
related to the displayed alert. The section’s table lists events that occurred during
the selected time span in chronological order, starting with the most recent event.To view a change event’s details, click the event’s row in the list to open its
Change event panel. This displays the event’s title, category, source, type,
time of occurrence, and label names and values. It also links to the alert’s source
and provides tabs to Comments and the change event entity’s JSON depiction.To further explore all listed change events, click Open explorer in the alert
details page’s Change events section. Clicking this link opens
Changes Explorer and populates it with the same
time range and query used to populate the alert details page’s list of change events.
The Alert information sidebar identifies the collection
identified as the alert’s Owner and its associated Team.
If the alert’s source is an SLO, this section also lists the SLO’s Runbook link
if one is defined.When designing collections and teams, add contextual links, details, and default
notification policies to ensure that responders can quickly identify and notify
responsible colleagues or follow established policies and processes when an alert
is triggered.The Additional information section of the sidebar lists any Annotations
defined on the alert’s triggering source. For example, if you defined
annotations or
templated links on a monitor or SLO,
the alert’s additional information lists each entry and populates any variables
defined in them. Link-valued annotations render as clickable links scoped to the
alert’s time range when the destination supports it.When designing something that can trigger an alert, such as a monitor or SLO, use
annotations to conditionally contextualize these alert views with information pertinent
to responders.
The sidebar includes Monitor information or SLO information, depending on
whether the alert was triggered by a monitor or
service level objective (SLO). This card shows the source’s current
status and other alerting signals active on that same source.
Current status: The overall state of the source monitor or SLO.
Status and Signal values: A table of other actively alerting signals from
the same source. The table excludes the signal for the alert you’re viewing. Each
Signal values entry links to that signal’s alert details page. Hover over an
entry to see its full label key/value pairs. A footer displays the count of other
alerting signals on the source, for example, 3 related alerts.
If the source has more than 10 other alerting signals, a Signal labels field
appears so you can filter the table to specific label key/value pairs.If no other signals are firing on the source, the card displays
No other alerting signals found for this monitor or
No other alerting signals found for this SLO.If the source monitor or SLO is unavailable, the card displays
Source monitor unavailable or Source SLO unavailable.Click Open details to open the source
monitor or
SLO detail page.
When an alert is resolved, document its resolution in the alert’s Resolution notes
section. These notes can provide context to identify recurring issues and capture
actions for incident reviews.
The alert details page’s Resolution notes section lists any resolution notes
associated with an alert.In addition to the note’s contents, each resolution note includes:
The user ID of the note’s author.
The date and time that the note was added.
An (edited) indicator, if the note was edited.
The note’s associated signals.
The date and time of the alert’s most recent resolution at the time the note was
added.
You can add multiple resolution notes to an alert. To add a resolution note from
an alert details page:
Click + Add to open the Create resolution note panel.
Enter the note in the Resolution note field. The field accepts Markdown formatting,
with shortcut buttons for Bold, Italics, code, and links.To preview formatting, enable the Show preview toggle.
Click Create to create the note, or Cancel to return to the alert details
page.
By default, the alert details page filters resolution notes by the actively firing
signal.Toggle Show current signal only off to include notes from all alert instances on
the same source. Duplicate notes can appear if the same note was added to more than
one instance.
