View and create

Use change events

Use change events to understand what changed and help investigate and remediate issues when they arise. You can create change events that don't already exist in the Chronosphere app.

You can also enable change events on Grafana dashboards by configuring an annotation with a change events query.

View change events

To view change events:

In the navigation menu select Explorers > Changes Explorer.
Click a change event in the results list and then click View More Details to display its full details.
Use the query box to filter change events to locate where and when an issue occurred.

Click Code deploys or Triggered alerts to run a predefined filter. The former filter returns all deploy change events that started or ended, including any feature flag changes. The latter filter returns all alert change events where a monitor triggered.
As you refine your query, click and select a portion of the time chart to zoom in on a shorter time window.

Change event details

Click a specific change event to view all of its available details in a tabbed interface.

Summary tab

The Summary tab includes an expanded view of the information from the main view, such as all labels associated with the change event. You can filter by label in this view by typing any part of the key or value for a particular label. Chronosphere automatically renders any links in the Labels section as hyperlinks, such as links to third-party sources, runbooks, or logs in your cloud provider.

JSON payload tab

The JSON Payload tab displays the JSON payload accompanying the selected change event. Use this view to investigate issues with a change event, share change event details with other on-call engineers, include snippets in a runbook, and use as a starting point for creating change events using the Chronosphere API. Explore this data to find metadata you want to extract and promote to the labels map, or data you need to remove from the labels map.

Create change events

Chronosphere automatically categorizes and displays change events in Changes Explorer to help on-call engineers understand what changed, which can expedite root cause analysis. However, you might want to explicitly create a change event, such as a broadcast to inform other users of an upcoming outage, or a third-party change event in addition to the predefined change event categories.

Use either the Chronosphere app or the Chronosphere API to create change events, and follow these best practices as guidelines.

To create change events with the Chronosphere app:

In the navigation menu select Explorers > Changes Explorer.
On the Changes Explorer page, click + Create Event.
In the Add Custom Event dialog, define your change event:
In the Title field, enter a descriptive, unique title for the change event.
Select a timestamp to associate with the change event. Event creation is limited to 24 hours in the past and 24 hours into the future from the current time.
Select the Category of change event, which is likely broadcast or third_party.
Select a Source for the change event, which defaults to chronosphere_ui.
Enter a Type that maps to the category you selected. For example, if you selected broadcasts as the category, you might enter outage as the event type.
Add event labels to associate with the change event as key/value pairs. Chronosphere requires the user_email and user_id keys, which you can't edit.
Click Add Event to create the change event.

Your event is viewable and searchable from Changes Explorer.

Enable change events

You can enable change events for charts on classic dashboards by adding annotations with a defined change events query. When enabled, change events display as a dotted-line annotation on your dashboard. You must enable annotations on each dashboard where you want them to display.

You can also use Grafana variable syntax (opens in a new tab) in your query, such as including $varname to interpolate values from your dashboard. Defining variables enables dropdown menus in dashboards to better filter your displayed data. In conjunction with annotations, you can display change events relevant to your other telemetry data such as metrics and traces. For example, if you filter your dashboard to display only metric data for your staging environment, you can display relevant change events for that environment only.

Limitations

The following limitations apply to change events for classic dashboards:

Change events created using the CreateEvent API (opens in a new tab) don't display as ranged annotations on Grafana charts. Only native change events, such as alerting monitors, and change events created through webhook integrations display as ranged annotations.
Change events display as annotations on Grafana graph panels only. If you use classic dashboards and want to display change events, your dashboard must use the graph panel type. Annotations aren't supported on other panel types, such as time series panels.

To enable change events on classic dashboard panels:

From a classic dashboard, click Settings > Advanced settings to display all dashboard options.
Optional: You can define variables and include them in your annotation. When the Grafana query evaluates, the current value replaces the variable.
1. Click the Variables tab, and then click New.
2. In the General section, enter a name for your variable in the Name field.
3. In the Query Options section, select a Refresh interval and enter your query in the Regex field. For example, you might create a variable named cluster with the following query:
```
label_values(frontend_requests, k8s_cluster)
```
4. In the Selection Options section, choose the options for how to display your data in the dashboard:
  - Multi-value: Presents each value for variable data as a checkbox in the variable dropdown menu to let you select multiple values.
  - Include all option: Provides an All option in the variable dropdown menu to let you select all available values.
  Refer to Grafana selection options for more information and examples about these options.
Click the Annotations tab, and then click Add Annotation Query.
In the Annotations page, enter a name for your annotation and select Chronosphere Timeline as the data source.
In the Query field, enter a change events query to define the data you want to display on your dashboard. If you defined variables, you can enter them in your query.

For example, the following query displays all change events categorized as deploys that contain an environment label including production-, and includes the $cluster variable you defined previously:
```
category = "deploys" AND labels.environment =~ "production-.*" AND title =~ ".*($cluster).*"
```
In the Include Labels field, enter a comma-separated list of labels you want to display in the annotation tooltip. If you enter a label that isn't mapped to an event, Chronosphere doesn't display that label in the annotation. For example:
```
deployment_id,target_service,user_email
```
Click Add to add your annotation, then click Save dashboard to save your changes.

Grafana selection options

When enabling change events for Grafana dashboards, you can define variables to include in your annotation. In the Selection Options section, you can choose Multi-value, Include all option, or both options. The approach you choose depends on your use case and user requirements, in addition to whether you integrate with other annotation plugins.

Multi-value: This option lets you select multiple values simultaneously, which is ideal for scenarios where you must associate annotations with various tags or categories.
Include all option: This option lets you select all values simultaneously. You can also enter a custom value to display annotations across all available values.

The following examples show how these settings operate for the Chronosphere Timeline data source. The query you construct and the end result differs depending on the data source you choose when defining a Grafana annotation.

Each of the following examples use the same query in the Query field of the Query Options section when defining a variable called cluster for a Grafana annotation:

label_values(production_request_ms_bucket, production_k8s_cluster)

Select neither option

If you select neither Multi-value or Include all option, the cluster dropdown menu in your classic dashboard lets you select only one value at a time.

Select include all with custom value

If you select Include all option and enter a wildcard (.*) value in the Custom all value field, the cluster dropdown menu in your Grafana dashboard includes an All option. Selecting All in the dropdown menu selects all values and is equivalent to the following query:

cluster =~ ".*.*.*"

Select multi-value only

If you select only Multi-value, you can select multiple values from the cluster dropdown menu in your classic dashboard. Selecting multiple values is equivalent to the following query, where production-a|production-b|... are the values:

cluster =~ “production-a|production-b|production-c|test-a|test-b”

Select both multi-value and include all

If you select both Multi-value and Include all option, and enter a wildcard (.*) value in the Custom all value field, the cluster dropdown menu in your classic dashboard both includes an All option and lets you select multiple values.

Selecting All in the dropdown menu is equivalent to the following query:

cluster =~ ".*.*.*"

Selecting multiple values in the dropdown menu is equivalent to the following query where production-a|production-b|... are the values:

cluster =~ “production-a|production-b|production-c|test-a|test-b”

Best practices

Use the following best practices when creating change events with the Chronosphere app and the Chronosphere CreateEvent API.

Categorize change events intentionally

How you categorize change events matters. Use category values to organize your change events. Chronosphere supports the following change event categories by default:

Category	Description
Alerts	Alerts are changes in telemetry caught by Chronosphere monitors and the event life cycle they use to track them. This includes alert triggers, resolved alerts, and alert notifications.
Broadcasts	Broadcasts are changes that track visibility for an outage, large marketing campaigns starting, or events that might cause an increase in usage such as the end of a sporting event or a big shopping day of the year.
Chronosphere	Chronosphere change events are changes to the configuration of the Chronosphere app that can affect the behavior of persisted data, evaluated by monitors and presented in dashboards.
Deploys	Deploys are changes that track the progress of new code deployed into an environment.
Feature flags	Feature flags change events are changes to feature flags and other dynamic configuration controls that affect the way code executes in a program without requiring a deployment.
Infrastructure	Infrastructure change events are changes to the underlying operating structures and fabric of an environment, such as networking, security, container orchestration, and data stores.
Third party	Third party change events are changes to the availability or status of dependencies outside of your organization, such as APIs your services rely on that other organizations provide and operate.

Chronosphere also supports custom change event categories. For example, you might rotate your secrets often, which can cause unintended authentication issues for users. To track those events, you might want to add a secrets_change category to quickly filter matching events. To request adding or removing a custom change event category in your tenant, contact your Technical Account Manger or Customer Success representative.

Identify the source

The primary objective when sending change events to Chronosphere is to maintain both high signal and low noise. Avoid sending low-value events that create noise.

The source field lets you know where events are coming from at all times. Use strings that help you identify the source instrumentation that's generating event data to help you locate that data.

Improve quality with proper typing

Types differentiate change events within a category or source. For example, you might have a source named deployer with a deploys category. You can use different types within that combination of source and category, such as deploy_start and deploy_end to identify when a deploy begins and ends. Types can help improve the quality of audit log events, and enable on-call engineers to better group and identify data anomalies at a given point in time.

Filter Events integrations