Monitor details

Monitor details

After you select an individual monitor, you can view all details for that monitor and filter query results to individual time series and related change events. You can also view a monitor's history of changes to understand when alerts triggered and which notifications sent.

If a monitor has any active muting rules, Chronosphere displays a banner at the top of the page indicating the number of active muting rules and when they expire.

Chronosphere displays a banner indicating Alerts for this monitor are being muted if a monitor is actively muted or was muted within the last hour.

View a monitor's history of changes

To view the history of changes for a selected monitor, click the three vertical dots icon and then click Version History to display a panel with two tabs:

  • Code Config: Displays a code representation of the selected monitor as of the time of the selected revision.
  • Code Diff: Displays a Git-style diff of the most-recent change made to the monitor, in Chronosphere API format.

To view any revision in the monitor's history, click it in the list of timestamped revisions. The timestamps default to your local timezone.

The Version History view retains up to 500 revisions, or up to 15 months of revisions if there are fewer than 500 revisions.

Review code representations in Code Config view

Select the code representation's format from the Config Mode dropdown. This view also provides Copy and Download buttons that respectively retrieve the selected version's code representation in the selected format to the clipboard or a downloaded file.

Inspect changes in Code Diff view

The Code Diff view displays the difference between the selected revision and another revision. By default, this compares the selected revision with the previous revision. To compare the selected revision to another revision in the history, click the Compare With dropdown and select the timestamp of the monitor revision that you want to compare.

You can view unchanged lines within the diff by clicking the Expand X lines... links.

To toggle between diff styles, select one of these options:

  • Unified: Displays changes stacked inline from a single-column view.
  • Split: Displays changes side-by-side with a two-column view.

Filter query results

When viewing an individual monitor, you can filter the results plotted on the graph. To display the query for the monitor, click the Show Query toggle.

  1. Change the displayed error state by clicking either All, All Alerting, or Critical Only.

  2. From the time picker menu, change the time range of the query results to modify the displayed time period.

  3. In the Series Legend, select one or more series in the list to filter the resulting time series that the graph displays. See Series legend for more information.

  4. Use the Change Events dropdown menu or advanced filtering syntax to display change events related to the selected monitor.

    You can create a change event such as a broadcast to inform other users of an upcoming outage, or a third-party change event in addition to the predefined change event categories.

  5. Click the Alert History tab to display a history of alerts generated by the monitor.

  6. Display additional monitor details by clicking the three vertical docs icon for the selected monitor:

    • Monitor Specifications includes details such as the data source, slug, and time interval for the monitor.

    • Conditions are tests that the results of a monitor query are evaluated against, with a related severity.

    • Notification Policies contain rules for notifications, which send to specified endpoints (such as PagerDuty, email, and Slack) when an alert triggers, and optionally when the alert resolves.

    • Edit Monitor opens a panel with tools for editing a monitor's configuration and notification policy, labels, query, signals, conditions, and annotations.

    • Version History opens the history of changes for the monitor. See View a monitor's history of changes for more information.

  7. To examine and modify the query and the metric the monitor uses, click Open in Explorer to open the query in the Metrics Explorer.

  8. To mute the selected monitor, a signal from a monitor, or any stored time series, add a muting rule.

Display change events

Use the Change Events dropdown menu or advanced filtering options to display change events on the Query Results panel for a monitor. Use basic filtering to select a predefined query that displays change events, or use advanced filtering to define a query if you know details about the change event you're searching for.

The following change event icons can display:

CategoryDescription
AlertsAlerts are changes in telemetry caught by Chronosphere monitors and the event life cycle they use to track them. This includes alert triggers, resolved alerts, and alert notifications.
BroadcastsBroadcasts are changes that track visibility for an outage, large marketing campaigns starting, or events that might cause an increase in usage such as the end of a sporting event or a big shopping day of the year.
Chronosphere ChronosphereChronosphere change events are changes to the configuration of the Chronosphere app that can affect the behavior of persisted data, evaluated by monitors and presented in dashboards.
DeploysDeploys are changes that track the progress of new code deployed into an environment.
Feature flagsFeature flags change events are changes to feature flags and other dynamic configuration controls that affect the way code executes in a program without requiring a deployment.
InfrastructureInfrastructure change events are changes to the underlying operating structures and fabric of an environment, such as networking, security, container orchestration, and data stores.
Third partyThird party change events are changes to the availability or status of dependencies outside of your organization, such as APIs your services rely on that other organizations provide and operate.

Basic filtering

From the Change Events dropdown menu, select one of the following predefined queries to display change events:

  • Broadcasts
  • Broadcasts, deploy events in production, and feature flag updates in production
  • Broadcasts, deploy events in production, feature flag updates in production, and alerts for the selected monitor
  • All events

Click any change event icon in the graph on the Query Results panel to open a more detailed view of change events for the selected time period. This list includes the event time, category, and title for each change event. Click any change event to view detailed information for a specific change event. When viewing this list, you can also click View in Changes Explorer to open the selected filter in Changes Explorer.

To learn more about change event details, refer to change event details.

Advanced filtering

If you know some of the criteria to search for, you can use advanced filtering to define a filter that displays specific change events. Use the lightweight filtering syntax to define which change events you want to display for the selected monitor.

The filtering syntax supports the following keys:

  • category: Event category, such as alerts, broadcasts, or deploys.
  • labels: Label metadata you've annotated your change events with, such as build, environment, or description.
  • source: Event source, such as alert_history, buildkite, or deployer.
  • title: Event title, either partial or complete.
  • type: Event type, such as build_started, deploy_start, or monitor_muted.

To use advanced filtering:

  1. Click Filter Events to display the filter box.
  2. Define a filter based on the change event attributes you want to filter on.

For example, the following filter returns all infrastructure change events where the build completed on the master branch:

category = "infrastructure" AND type = "build_finished" AND labels.build.branch = "master"

Filters can span multiple lines, and you can use AND plus OR operators to include or exclude parameters in your query.

  1. Click Filter, or press Shift+Enter (Shift+Return on macOS) to run the query.

  2. In the query result histogram, click any change event icon to view more information about each change event.

  3. When viewing change event details, click a change event from the list to view more detailed information.

    To learn more about change event details, refer to change event details.

Create a change event

You can create a change event directly from the details page for a specific monitor:

  1. On the monitor details page, click + Create Event.

  2. In the Add Custom Event dialog, define your change event:

  3. In the Title field, enter a descriptive, unique title for the change event.

  4. Select a timestamp to associate with the change event. Event creation is limited to 24 hours in the past and 24 hours into the future from the current time.

  5. Select the Category of change event, which is likely broadcast or third_party.

  6. Select a Source for the change event, which defaults to chronosphere_ui.

  7. Add event labels to associate with the change event as key/value pairs. Chronosphere requires the user_email and user_id keys, which you can't edit.

  8. Enter a Type that maps to the category you selected. For example, if you selected broadcasts as the category, you might enter outage as the event type.

  9. Click Add Event to create the change event.

Series legend

The series legend displays labels for all metrics displayed in the graph as either a list or table view.

Both views display the label keys and values for each series and include the current alert status (clear, warning, critical).

You can filter the list of the resulting time series with the Search Series search box.

Click an item in the list to isolate the related line on the graph. Clicking the item again clears the selection. You can Control+click (Command+click on macOS) to choose multiple items.

Annotations

The annotations defined for the monitor, such as runbooks, links to related dashboards, data links to related traces, and documentation links. See Annotations for more information.

Alert history

The Alert History tab next to Monitor Info displays a history of alerts generated by the monitor.

You can change the order of the history by clicking the Timestamp (UTC) column header, and then clicking the chevron to toggle display of the JSON payload for an alert.

The page also lets you filter the history by event type, or toggle the scope of the Alert history between the currently selected signal or all signals.

Valid event types are:

  • Alert triggered
  • Alert resolved
  • Notification sent (alert triggered)
  • Notification failed (alert triggered)
  • Notification sent (alert resolved)
  • Notification failed (alert resolved)
  • Alert muted
  • Alert unmuted

There can be up to a five minute delay between the time an alert for a monitor resolves (Alert resolved) and the time when a notification sends indicating the alert resolved (Notification sent (alert resolved)).