Alert details

Chronosphere Observability Platform lists all triggered alerts and their status, and generates detailed visualizations of each alert. The visualizations and related tools help you analyze when, why, and how an alert was triggered. Each alert instance is identified by the unique combination of series labels and static entity labels that remain after any signal grouping is applied. This means two triggered alerts are distinct instances when their label sets differ, even if they come from the same monitor. Each alert instance has a unique URL. Click Copy URL in the page header to copy a shareable URL to your clipboard. Share it with teammates so they can view the same page directly. The alert details page is mobile-responsive so you can view alert status, examine chart data, and mute alerts from a mobile device.

​

View the list of alerts

To view the list of triggered alerts, select Alerting > Alerts in the navigation menu. Each listed alert includes several columns of information:

• The date and time that it was Created at
• Its Status, designated with an icon, text, and color (see Alert status)
• Its title (Alert), as a link to its alert details page
• The duration that the alert has been Alerting for
• The Configuration entity related to the alert, as a link to the associated entity, such as a monitor or SLO

You can define the number of Rows per page to display by clicking its dropdown and selecting a value. To navigate between pages, click the < Go to previous page and > Go to next page buttons.

​

Alert status

The alerts list and alert details page designate status with an icon, text, and color:

​

View an alert’s details

Access an alert’s detail page from the alerts list or wherever a triggered alert is referenced, including from the source entity, such as the monitor or service level objective (SLO), that defined the alert’s triggers.

​

Status and signals

Each alert presents alert details in a dedicated section. This section includes the alert’s status, a visualization of the triggering queries and change events, and a table of the series and conditions that triggered the alert. For status icon and label descriptions, see Alert status. The page also lists any triggering Signal as chips that identify its keys and their values.

​

Query tabs

The alert details section provides two tabs for viewing query results:

• Stored data: Default. Visualizes the alert’s original query of currently stored metric data. Results typically match what the alerting engine evaluated, but can differ if late-arriving data has since been ingested. To account for ingestion delays, consider adding an offset to your query.
• Evaluated data: Queries the ALERTS_VALUE metric, which records the values the alerting engine computed at each evaluation interval. See Alert metrics for reference documentation about the ALERTS metric and its labels. This view shows what the monitor actually saw when it made its alerting decision, regardless of data that arrived later. The table for this tab includes alertstate and severity columns. A series with alertstate set to pending triggers an alert after it has been continuously breaching the threshold for the sustain duration configured on the monitor.

Use the Evaluated data tab to determine if the alerting engine observed data differently than what’s currently stored, such as when late-arriving metrics change the stored view. Both tabs display query results as a time series chart. You can optionally:

• Toggle Show thresholds, which draws dotted horizontal lines on the time series chart depicting the alert’s triggering thresholds.
• Toggle Show query or Show queries, which displays the active tab’s underlying queries.
• Use the three vertical dots icon to perform additional actions, including:
  • Open in explorer, which opens Metrics Explorer and populates it with the active tab’s query.
  • Click Analyze anomaly (DDx) to perform differential diagnosis on the data.
• Hold the pointer over the chart to display an additional three vertical dots icon with additional common panel actions.

The Open in explorer and Show query controls reflect whichever tab is active. When viewing the Evaluated data tab, the explorer link and query preview display the ALERTS_VALUE query. When viewing the Stored data tab, they display the alert’s source query.

​

Late-arriving data notice

A warning banner can display with the alert’s time series chart to indicate that late-arriving data might have affected the alert. This banner appears on the Stored data tab, and on the Availability tab for SLO alerts, in the following situations:

• The firing alert’s data is delayed: For an actively firing alert, the most recent data point for the firing series is older than the rate at which the data usually updates. The banner reports how old the latest data point is and the expected update cadence, and cautions that the alert might have triggered on incomplete data. Check the alert again in a few minutes, after the data catches up.
• Stored data no longer meets the triggering condition: The data currently stored for when the alert started firing no longer meets the condition that triggered it, because late-arriving data changed the result after the monitor or SLO evaluated. Use the Evaluated data tab to see the values that were recorded when the alert triggered.

If either situation happens regularly, add an offset to your query to account for ingestion delays. For more information, see Add offsets to your query.

​

Chart and series tables

In the time series table, select from visualized series to highlight them in the visualization, and you can use the Search series query box to filter the list. You can also toggle between two views. The default table view, lists the time series’ status as an icon and its labels and their values. The list view lists only each series’ status and the query that produces it. You can optionally reveal the Conditions that triggered the alert. Observability Platform lists these conditions as a table of each condition’s status, operator, sustained duration, time to resolution, and the signals to which the alert’s conditions apply.

​

SLO panels

If the alert’s source is an SLO, the alert details page also reproduces the SLO’s SLI breakdown and Burn/Error rates sections for reference. For more information, see Service level objectives.

​

Mute an alert

To mute an alert, create a muting rule. You can do this directly from an alert details page by clicking the Mute alert button. This opens a panel to create a muting rule that’s populated with the alert’s relevant source and name, and without requiring you to leave the alert details page. If a muting rule is already muting a triggered alert, clicking this button instead opens a panel to edit the associated muting rule. Alternatively, you can select Alerting > Muting Rules in the navigation menu to create new muting rules or edit existing muting rules. However, doing so doesn’t populate the muting rule with details from a specific alert’s source.

​

Define the alert detail view’s time range

You can customize the chronological scope of an alert’s details by selecting a time range, which defaults to the Last 1 hour. The chart and time series table update to depict time series data only within the selected time range, but the alert’s status always displays its current state.

​

View an alert’s history

The Alert history section on the alert details page lists past instances of the same alert. Past instances are previous occurrences with the same signal labels from the same monitor or SLO. Each past instance appears as a link labeled with the relative time it was created, such as 2 hours ago. Click any link to navigate to that instance’s alert details page. The current instance is labeled with the relative time followed by (This instance) and can be expanded to show the events associated with it. If more recent instances exist outside the initially loaded window, a Show more recent alerts button appears at the top of the list.

​

Navigate to an alert’s source

Each alert detail page includes a link to return to the source of its trigger. For instance, an alert triggered by a monitor has a Return to source monitor link, and an alert triggered by an SLO has a Return to source SLO link. If you navigate to an alert’s source, you can return to the alert either by clicking the triggered alert in the source entity. You can also return to the alerts list and filter it by the alert’s title.

​

Investigate an alert with the Assistant

The Assistant reads triggered and resolved alerts, and explains what changed around one. Because it knows which alert page you’re viewing, you can ask about the open alert without identifying it. To investigate the open alert, click Ask Assistant in the page header or press A, and then describe what you want to know. For example: The Assistant correlates the alert’s signal against logs, traces, and change events, and can start a deep investigation that runs while you continue triaging. See Start an investigation.

​

Examine an alert’s related change events

If you’ve enabled change events, the alert details page displays associated events in the time series chart and lists them in the Change events section. To configure which change events appear in the chart, click Events to open the Display events panel. Under Change events:

• Toggle Show event markers to show or hide event markers on the chart.
• Use the Code or Builder mode toggle to configure the change events query. In Builder mode, use the category table to select which event categories to display.

Click Save to apply your changes. The change events list includes each Alert change event related to the displayed alert. The section’s table lists events that occurred during the selected time span in chronological order, starting with the most recent event. To view a change event’s details, click the event’s row in the list to open its Change event panel. This displays the event’s title, category, source, type, time of occurrence, and label names and values. It also links to the alert’s source and provides tabs to Comments and the change event entity’s JSON depiction. To further explore all listed change events, click Open explorer in the alert details page’s Change events section. Clicking this link opens Changes Explorer and populates it with the same time range and query used to populate the alert details page’s list of change events.

​

View an alert’s related information

The Alert information sidebar identifies the collection identified as the alert’s Owner and its associated Team. If the alert’s source is an SLO, this section also lists the SLO’s Runbook link if one is defined. When designing collections and teams, add contextual links, details, and default notification policies to ensure that responders can quickly identify and notify responsible colleagues or follow established policies and processes when an alert is triggered. The Additional information section of the sidebar lists any Annotations defined on the alert’s triggering source. For example, if you defined annotations or templated links on a monitor or SLO, the alert’s additional information lists each entry and populates any variables defined in them. Link-valued annotations render as clickable links scoped to the alert’s time range when the destination supports it. When designing something that can trigger an alert, such as a monitor or SLO, use annotations to conditionally contextualize these alert views with information pertinent to responders.

​

Monitor or SLO information

The sidebar includes Monitor information or SLO information, depending on whether the alert was triggered by a monitor or service level objective (SLO). This card shows the source’s current status and other alerting signals active on that same source.

• Current status: The overall state of the source monitor or SLO.
• Status and Signal values: A table of other actively alerting signals from the same source. The table excludes the signal for the alert you’re viewing. Each Signal values entry links to that signal’s alert details page. Hover over an entry to see its full label key-value pairs. A footer displays the count of other alerting signals on the source, for example, 3 related alerts.

If the source has more than 10 other alerting signals, a Signal labels field appears so you can filter the table to specific label key-value pairs. If no other signals are firing on the source, the card displays No other alerting signals found for this monitor or No other alerting signals found for this SLO. If the source monitor or SLO is unavailable, the card displays Source monitor unavailable or Source SLO unavailable. Click Open details to open the source monitor or SLO detail page.

​

Document an alert’s resolution

When an alert is resolved, document its resolution in the alert’s Resolution notes section. These notes can provide context to identify recurring issues and capture actions for incident reviews.

​

View resolution notes

The alert details page’s Resolution notes section lists any resolution notes associated with an alert. In addition to the note’s contents, each resolution note includes:

• The user ID of the note’s author.
• The date and time that the note was added.
• An (edited) indicator, if the note was edited.
• The note’s associated signals.
• The date and time of the alert’s most recent resolution at the time the note was added.

You can filter the list of resolution notes, edit a note, or delete a note.

​

Add a resolution note

You can add multiple resolution notes to an alert. To add a resolution note from an alert details page:

1. Click + Add to open the Create resolution note panel.
2. Enter the note in the Resolution note field. The field accepts Markdown formatting, with shortcut buttons for Bold, Italics, code, and links. To preview formatting, enable the Show preview toggle.
3. Click Create to create the note, or Cancel to return to the alert details page.

​

Edit a resolution note

To edit a resolution note:

1. Click Edit resolution note for the note you want to edit.
2. Edit the resolution note’s contents.
3. Click Save to save the changes, or Cancel to return to the alert details page.

​

Delete a resolution note

To delete a resolution note:

1. Click Delete resolution note for the note you want to delete.
2. Click Delete to confirm that you want to delete the note.

​

Filter resolution notes

You can filter resolution notes by signal or by text.

​

Filter notes by signal

By default, the alert details page filters resolution notes by the actively firing signal. Toggle Show current signal only off to include notes from all alert instances on the same source. Duplicate notes can appear if the same note was added to more than one instance.

​

Filter notes by text

To filter resolution notes by their contents:

1. On the alert details page, click the Search resolution notes field.
2. Enter text to display only the resolution notes that contain that text.