Alert details

Chronosphere Observability Platform generates detailed visualizations of each triggered alert, and provides tools to help you analyze when, why, and how the alert was triggered. You can access alert detail pages wherever a triggered alert is referenced, including:

• From the list of alerts, which you can access by selecting Alerting > Alerts in the navigation menu.
• From the source entity, such as the monitor or service level objective (SLO), that defined the alert’s triggers

Each alert instance has a unique URL. Click Copy URL in the page header to copy a shareable URL to your clipboard. Share it with teammates so they can view the same page directly. The alert details page is mobile-responsive so you can view alert status, examine chart data, and mute alerts from a mobile device.

​

View an alert’s details

Each alert presents alert details in a dedicated section. This section includes the alert’s status, a visualization of the triggering queries and change events, and a table of the series and conditions that triggered the alert. The page designates the alert’s status with an icon, text, and color: The page also lists any triggering Signal as chips that identify its keys and their values. The alert details section provides two tabs for viewing query results:

• Stored data: Default. Visualizes the alert’s original query of currently stored metric data. Results typically match what the alerting engine evaluated, but can differ if late-arriving data has since been ingested. To account for ingestion delays, consider adding an offset to your query.
• Evaluated data: Queries the ALERTS_VALUE metric, which records the values the alerting engine computed at each evaluation interval. This view shows what the monitor actually saw when it made its alerting decision, regardless of data that arrived later. The table for this tab includes alertstate and severity columns. A series with alertstate set to pending triggers an alert after it has been continuously breaching the threshold for the sustain duration configured on the monitor.

Use the Evaluated data tab to determine if the alerting engine observed data differently than what’s currently stored, such as when late-arriving metrics change the stored view. Both tabs display query results as a time series chart. You can optionally:

• Show thresholds, which draws dotted horizontal lines on the time series chart depicting the alert’s triggering thresholds.
• Show query, which displays the active tab’s underlying query.
• Perform differential diagnosis (DDx) on the data.
• Open in explorer, which opens Metrics Explorer and populates it with the active tab’s query.
• Add to notebook. See Notebooks to add the query results chart from the panel menu.

The Open in explorer and Show query controls reflect whichever tab is active. When viewing the Evaluated data tab, the explorer link and query preview display the ALERTS_VALUE query. When viewing the Stored data tab, they display the alert’s source query. In the time series table, select from visualized series to highlight them in the visualization, and you can use the Search series query box to filter the list. You can also toggle between two views. The default table view, lists the time series’ status as an icon and its labels and their values. The list view lists only each series’ status and the query that produces it. You can optionally reveal the Conditions that triggered the alert. Observability Platform lists these conditions as a table of each condition’s status, operator, sustained duration, time to resolution, and the signals to which the alert’s conditions apply.

​

SLO panels

If the alert’s source is an SLO, the alert details page also reproduces the SLO’s SLI breakdown and Burn/Error rates sections for reference. For more information, see Service level objectives.

​

Document an alert’s resolution

When an alert is resolved, document its resolution in the alert’s Resolution notes section. These notes can provide context to identify recurring issues and capture actions for incident reviews.

​

View resolution notes

The alert details page’s Resolution notes section lists any resolution notes associated with an alert. In addition to the note’s contents, each resolution note includes:

• The user ID of the note’s author.
• The date and time that the note was added.
• An (edited) indicator, if the note was edited.
• The note’s associated signals.
• The date and time of the alert’s most recent resolution at the time the note was added.

You can filter the list of resolution notes, edit a note, or delete a note.

​

Add a resolution note

You can add multiple resolution notes to an alert. To add a resolution note from an alert details page:

1. Click + Add to open the Create resolution note panel.
2. Enter the note in the Resolution note field. The field accepts Markdown formatting, with shortcut buttons for Bold, Italics, code, and links. To preview formatting, enable the Show preview toggle.
3. Click Create to create the note, or Cancel to return to the alert details page.

​

Edit a resolution note

To edit a resolution note:

1. Click Edit resolution note for the note you want to edit.
2. Edit the resolution note’s contents.
3. Click Save to save the changes, or Cancel to return to the alert details page.

​

Delete a resolution note

To delete a resolution note:

1. Click Delete resolution note for the note you want to delete.
2. Click Delete to confirm that you want to delete the note.

​

Filter resolution notes

You can filter resolution notes by signal or by text.

​

Filter notes by signal

By default, the alert details page filters resolution notes by the actively firing signal. Toggle Show current signal only off to include notes from all alert instances on the same source. Duplicate notes can appear if the same note was added to more than one instance.

​

Filter notes by text

To filter resolution notes by their contents:

1. On the alert details page, click the Search resolution notes field.
2. Enter text to display only the resolution notes that contain that text.

​

Mute an alert

To mute an alert, create a muting rule. You can do this directly from an alert details page by clicking the Mute alert button. This opens a panel to create a muting rule that’s populated with the alert’s relevant source and name, and without requiring you to leave the alert details page. If a muting rule is already muting a triggered alert, clicking this button instead opens a panel to edit the associated muting rule. Alternatively, you can select Alerting > Muting Rules in the navigation menu to create new muting rules or edit existing muting rules. However, doing so doesn’t populate the muting rule with details from a specific alert’s source.

​

Navigate to an alert’s source

Each alert detail page includes a link to return to the source of its trigger. For instance, an alert triggered by a monitor has a Return to source monitor link, and an alert triggered by an SLO has a Return to source SLO link. If you navigate to an alert’s source, you can return to the alert either by clicking the triggered alert in the source entity. You can also navigate to the list of alerts and filter it by the alert’s title.

​

View an alert’s history

The Alert history section on the alert details page lists past instances of the same alert. Past instances are previous occurrences with the same signal labels from the same monitor or SLO. Each past instance appears as a link labeled with the relative time it was created, such as 2 hours ago. Click any link to navigate to that instance’s alert details page. The current instance is labeled with the relative time followed by (This instance) and can be expanded to show the events associated with it. If more recent instances exist outside the initially loaded window, a Show more recent alerts button appears at the top of the list.

​

Define the alert detail view’s time range

You can customize the chronological scope of an alert’s details by selecting a time range, which defaults to the Last 1 hour. The chart and time series table update to depict time series data only within the selected time range, but the alert’s status always displays its current state.

​

Perform differential diagnosis (DDx) on an alert

You can also use differential diagnosis (DDx) from an alert’s detail page. This helps you identify correlations within specific alerts while investigating the root causes of their triggers.

​

Examine an alert’s related change events

If you’ve enabled change events, the alert details page displays associated events in the time series chart and lists them in the Change events section. To configure which change events appear in the chart, click Events to open the Display events panel. Under Change events:

• Toggle Show event markers to show or hide event markers on the chart.
• Use the Code or Builder mode toggle to configure the change events query. In Builder mode, use the category table to select which event categories to display.

Click Save to apply your changes. The change events list includes each Alert change event related to the displayed alert. The section’s table lists events that occurred during the selected time span in chronological order, starting with the most recent event. To view a change event’s details, click the event’s row in the list to open its Change event panel. This displays the event’s title, category, source, type, time of occurrence, and label names and values. It also links to the alert’s source and provides tabs to Comments and the change event entity’s JSON depiction. To further explore all listed change events, click Open explorer in the alert details page’s Change events section. Clicking this link opens Changes Explorer and populates it with the same time range and query used to populate the alert details page’s list of change events.

​

View an alert’s related information

The Alert information sidebar identifies the collection identified as the alert’s Owner and its associated Team. If the alert’s source is an SLO, this section also lists the SLO’s Runbook link if one is defined. When designing collections and teams, add contextual links, details, and default notification policies to ensure that responders can quickly identify and notify responsible colleagues or follow established policies and processes when an alert is triggered. The Additional information section of the sidebar lists any Annotations defined on the alert’s triggering source. For example, if you defined annotations on a monitor, the alert’s additional information lists each of the monitor’s annotations and populates any variables defined in them. When designing something that can trigger an alert, such as a monitor or SLO, use annotations to conditionally contextualize these alert views with information pertinent to responders.

​

Monitor or SLO information

The sidebar includes Monitor information or SLO information, depending on whether the alert was triggered by a monitor or service level objective (SLO). This card shows the source’s current status and other alerting signals active on that same source.

• Current status: The overall state of the source monitor or SLO.
• Status and Signal values: A table of other actively alerting signals from the same source. The table excludes the signal for the alert you’re viewing. Each Signal values entry links to that signal’s alert details page. Hover over an entry to see its full label key/value pairs. A footer displays the count of other alerting signals on the source, for example, 3 related alerts.

If the source has more than 10 other alerting signals, a Signal labels field appears so you can filter the table to specific label key/value pairs. If no other signals are firing on the source, the card displays No other alerting signals found for this monitor or No other alerting signals found for this SLO. If the source monitor or SLO is unavailable, the card displays Source monitor unavailable or Source SLO unavailable. Click Open details to open the source monitor or SLO detail page.