View and create

Use change events

Use change events to understand what changed and help investigate and remediate issues when they arise. You can create change events that don't already exist in Chronosphere Observability Platform.

You can also enable change events on standard and classic dashboards by configuring an annotation with a change events query.

View change events

Select from the following methods to view change events.

To view change events:

In the navigation menu select Explorers > Changes Explorer.
Click a change event in the results list and then click View More Details to display its full details.
Use the query box to filter change events to locate where and when an issue occurred.

Click Code deploys or Triggered alerts to run a predefined filter. The former filter returns all deploy change events that started or ended, including any feature flag changes. The latter filter returns all alert change events where a monitor triggered.
As you refine your query, click and select a portion of the time chart to zoom in on a shorter time window.

Change event details

Click a specific change event to view all of its available details in a tabbed interface.

Summary tab

The Summary tab includes an expanded view of the information from the main view, such as all labels associated with the change event. You can filter by label in this view by typing any part of the key or value for a particular label. Observability Platform automatically renders any links in the Labels section as hyperlinks, such as links to third-party sources, runbooks, or logs in your cloud provider.

JSON payload tab

The JSON Payload tab displays the JSON payload accompanying the selected change event. Use this view to investigate issues with a change event, share change event details with other on-call engineers, include snippets in a runbook, and use as a starting point for creating change events using the Chronosphere API. Explore this data to find metadata you want to extract and promote to the labels map, or data you need to remove from the labels map.

Create change events

Observability Platform automatically categorizes and displays change events in Changes Explorer to help on-call engineers understand what changed, which can expedite root cause analysis. However, you might want to explicitly create a change event, such as a broadcast to inform other users of an upcoming outage, or a third-party change event in addition to the predefined change event categories.

Use either Observability Platform or the Chronosphere API to create change events, and follow these best practices as guidelines.

To create change events with Observability Platform:

In the navigation menu select Explorers > Changes Explorer.
On the Changes Explorer page, click + Create Event.
In the Add Custom Event dialog, define your change event:
In the Title field, enter a descriptive, unique title for the change event.
Select a timestamp to associate with the change event. Event creation is limited to 24 hours in the past and 24 hours into the future from the current time.
Select the Category of change event, which is likely broadcast or third_party.
Select a Source for the change event, which defaults to chronosphere_ui.
Enter a Type that maps to the category you selected. For example, if you selected broadcasts as the category, you might enter outage as the event type.
Add event labels to associate with the change event as key/value pairs. Observability Platform requires the user_email and user_id keys, which you can't edit.
Click Add Event to create the change event.

Your event is viewable and searchable from Changes Explorer.

Best practices

Use the following best practices when creating change events with Observability Platform and the Chronosphere CreateEvent API.

Categorize change events intentionally

How you categorize change events matters. Use category values to organize your change events. Chronosphere supports the following change event categories by default:

Category	Description
Alerts	Alerts are changes in telemetry caught by Observability Platform monitors and the event life cycle they use to track them. This includes alert triggers, resolved alerts, and alert notifications.
Broadcasts	Broadcasts are changes that track visibility for an outage, large marketing campaigns starting, or events that might cause an increase in usage such as the end of a sporting event or a big shopping day of the year.
Chronosphere	Observability Platform change events are changes to the configuration of Observability Platform that can affect the behavior of persisted data, evaluated by monitors and presented in dashboards.
Deploys	Deploys are changes that track the progress of new code deployed into an environment.
Feature flags	Feature flags change events are changes to feature flags and other dynamic configuration controls that affect the way code executes in a program without requiring a deployment.
Infrastructure	Infrastructure change events are changes to the underlying operating structures and fabric of an environment, such as networking, security, container orchestration, and data stores.
Third party	Third party change events are changes to the availability or status of dependencies outside of your organization, such as APIs your services rely on that other organizations provide and operate.

Chronosphere also supports custom change event categories. For example, you might rotate your secrets often, which can cause unintended authentication issues for users. To track those events, you might want to add a secrets_change category to quickly filter matching events. To request adding or removing a custom change event category in your tenant, contact Chronosphere Support.

Identify the source

The primary objective when sending change events to Observability Platform is to maintain both high signal and low noise. Avoid sending low-value events that create noise.

The source field lets you know where events are coming from at all times. Use strings that help you identify the source instrumentation that's generating event data to help you locate that data.

Improve quality with proper typing

Types differentiate change events within a category or source. For example, you might have a source named deployer with a deploys category. You can use different types within that combination of source and category, such as deploy_start and deploy_end to identify when a deploy begins and ends. Types can help improve the quality of audit log events, and enable on-call engineers to better group and identify data anomalies at a given point in time.

Filter Enable events