OBSERVABILITY PLATFORM
Monitor details

Monitor details

After you select an individual monitor, you can view all details for that monitor and filter query results to individual time series and related change events. You can also view a monitor's history of changes to understand when alerts triggered and which notifications sent.

If a monitor has any active muting rules, Chronosphere displays a banner at the top of the page indicating the number of active muting rules and when they expire.

Chronosphere displays a banner indicating Alerts for this monitor are being muted if a monitor is actively muted or was muted within the last hour.

View a monitor's history of changes

To view the history of changes for a selected monitor, click the three vertical dots icon and then click Version history to display a panel with two tabs:

  • Code config: Displays a code representation of the selected monitor as of the time of the selected revision.
  • Code diff: Displays a Git-style diff of the most-recent change made to the monitor, in Chronosphere API format.

To view any revision in the monitor's history, click it in the list of timestamped revisions. The timestamps default to your local timezone.

The Version History view retains up to 500 revisions, or up to 15 months of revisions if there are fewer than 500 revisions.

Review code representations in Code config view

Select the code representation's format from the Config Mode dropdown. This view also provides Copy and Download buttons that respectively retrieve the selected version's code representation in the selected format to the clipboard or a downloaded file.

Inspect changes in Code diff view

The Code diff view displays the difference between the selected revision and another revision. By default, this compares the selected revision with the previous revision. To compare the selected revision to another revision in the history, click the Compare With dropdown and select the timestamp of the monitor revision that you want to compare.

You can view unchanged lines within the diff by clicking the Expand X lines links.

To toggle between diff styles, select one of these options:

  • Unified: Displays changes stacked inline from a single-column view.
  • Split: Displays changes side-by-side with a two-column view.

Filter query results

When viewing an individual monitor, you can filter the results plotted on the graph. To display the query for the monitor, click the Show Query toggle.

  1. Change the displayed error state by clicking either All, All Alerting, or Critical Only.

  2. From the time picker menu, change the time range of the query results to modify the displayed time period.

  3. In the Series Legend, select one or more series in the list to filter the resulting time series that the graph displays. See Series legend for more information.

  4. Use the Change Events dropdown menu or advanced filtering syntax to display change events related to the selected monitor.

    You can create a change event such as a broadcast to inform other users of an upcoming outage, or a third-party change event in addition to the predefined change event categories.

  5. Click the Alert History tab to display a history of alerts generated by the monitor.

  6. Display additional monitor details by clicking the three vertical docs icon for the selected monitor:

    • Monitor Specifications includes details such as the data source, slug, and time interval for the monitor.

    • Conditions are tests that the results of a monitor query are evaluated against, with a related severity.

    • Notification Policies contain rules for notifications, which send to specified endpoints (such as PagerDuty, email, and Slack) when an alert triggers, and optionally when the alert resolves.

    • Edit Monitor opens a panel with tools for editing a monitor's configuration and notification policy, labels, query, signals, conditions, and annotations.

    • Version History opens the history of changes for the monitor. See View a monitor's history of changes for more information.

  7. To examine and modify the query and the metric the monitor uses, click Open in Explorer to open the query in the Metrics Explorer.

  8. To mute the selected monitor, a signal from a monitor, or any stored time series, add a muting rule.

Display change events

Use the Change Events dropdown menu or advanced filtering options to display change events on the Query Results panel for a monitor. Use basic filtering to select a predefined query that displays change events, or use advanced filtering to define a query if you know details about the change event you're searching for.

Change events use category icons for ease of visibility into different event types.

Basic filtering

From the Change Events dropdown menu, select one of the following predefined queries to display change events:

  • Broadcasts
  • Broadcasts, deploy events in production, and feature flag updates in production
  • Broadcasts, deploy events in production, feature flag updates in production, and alerts for the selected monitor
  • All events

Click any change event icon in the graph on the Query Results panel to open a more detailed view of change events for the selected time period. This list includes the event time, category, and title for each change event. Click any change event to view detailed information for a specific change event. When viewing this list, you can also click View in Changes Explorer to open the selected filter in Changes Explorer.

To learn more about change event details, refer to change event details.

Advanced filtering

If you know some of the criteria to search for, you can use advanced filtering to define a filter that displays specific change events. Use the lightweight filtering syntax to define which change events you want to display for the selected monitor.

The filtering syntax supports the following keys:

  • category: Event category, such as alerts, broadcasts, or deploys.
  • labels: Label metadata you've annotated your change events with, such as build, environment, or description.
  • source: Event source, such as alert_history, buildkite, or deployer.
  • title: Event title, either partial or complete.
  • type: Event type, such as build_started, deploy_start, or monitor_muted.

To use advanced filtering:

  1. Click Filter Events to display the filter box.
  2. Define a filter based on the change event attributes you want to filter on.

For example, the following filter returns all infrastructure change events where the build completed on the master branch:

category = "infrastructure" AND type = "build_finished" AND labels.build.branch = "master"

Filters can span multiple lines, and you can use AND plus OR operators to include or exclude parameters in your query.

  1. Click Filter, or press Shift+Enter (Shift+Return on macOS) to run the query.

  2. In the query result histogram, click any change event icon to view more information about each change event.

  3. When viewing change event details, click a change event from the list to view more detailed information.

    To learn more about change event details, refer to change event details.

Create a change event

You can create a change event directly from the details page for a specific monitor:

  1. On the monitor details page, click + Create Event.
  2. Follow the instructions to create a change event.

Series legend

The series legend displays labels for all metrics displayed in the graph as either a list or table view.

Both views display the label keys and values for each series and include the current alert status (clear, warning, critical).

You can filter the list of the resulting time series with the Search Series search box.

Click an item in the list to isolate the related line on the graph. Clicking the item again clears the selection. You can Control+click (Command+click on macOS) to choose multiple items.

Annotations

The annotations defined for the monitor, such as runbooks, links to related dashboards, data links to related traces, and documentation links. See Annotations for more information.

Alert history

The Alert History tab next to Monitor Info displays a history of alerts generated by the monitor.

You can change the order of the history by clicking the Timestamp (UTC) column header, and then clicking the chevron to toggle display of the JSON payload for an alert.

The page also lets you filter the history by event type, or toggle the scope of the Alert history between the currently selected signal or all signals.

There can be up to a five minute delay between the time an alert for a monitor resolves (Alert resolved) and the time when a notification sends indicating the alert resolved (Notification sent (alert resolved)).

Alert event payload

The Alert History tab displays the values captured when the alert fired. The payload fields are primarily defined in the monitor data model.

  • monitorSlug: The monitor's unique slug.
  • eventType: The type of event triggered. Valid alert types are:
    • Alert triggered
    • Alert resolved
    • Notification sent (alert triggered)
    • Notification failed (alert triggered)
    • Notification sent (alert resolved)
    • Notification failed (alert resolved)
    • Alert muted
    • Alert unmuted
  • createdAt: The time the alert fired.
  • signal: See signals. Contains one or more name and value.
  • details: Contains severity as defined by the notification policy and notifier. notifier values contain the notifier name and slug.