Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt

Use this file to discover all available pages before exploring further.

Monitors use a custom definition in YAML format. When adding or editing a monitor in Chronosphere Observability Platform, click the Code Config tool to view code representations of a monitor for Terraform, Chronoctl, and the Chronosphere API. Whichever method you use to create a monitor, the monitor structure uses the following data model. A full list of values is available in the API.

​
Collection

A collection is an organization of resources associated with a team. All monitors must belong to exactly one collection. Each team can own many collections, which are a group of resources like monitors and dashboards. The identified team owns the systems that the associated dashboards and monitors track. Each collection can have either a default notification policy for all monitors, or you can set notification policies explicitly for each monitor. These policies apply a function to time series data that determines whether an alert triggers.

​
Name

A unique descriptive name for the monitor. You can change a monitor’s name after it’s created. Monitor names are static strings, so they can’t include label variables such as $labels.LABEL_NAME.

​
Slug

A unique identifier that you can’t change after creating the monitor. Observability Platform generates a slug when you create a monitor. Slugs have a maximum length of 254 alphanumeric characters.

​
Notification policy

The notification policy field determines which notification policy to use at a particular alert severity. You define the policies in a collection. When you create a collection with Chronoctl or Terraform, you can reference a notification policy in any collection.

​
Notification template

This feature isn’t available to all Chronosphere Observability Platform users and might not be visible in your app. For information about enabling this feature in your environment, contact Chronosphere Support.
You can optionally customize the title and description of notifications when this monitor’s alerts trigger or resolve. Set a notification_template object on the monitor with title and description template strings. Observability Platform evaluates those strings at notification time using alert context, with Go text/template syntax. Configure the field through the Chronosphere API, Chronoctl, or Terraform. If a monitor sets a notification_template, it overrides any template information in the notifier to which the notification is sent. For variables, functions, and examples, see Notification templates. For the exact field shapes, see the monitor resource on CreateMonitor. In Chronoctl YAML, notification_template exists in the spec section of the monitor. The same field names apply in Terraform and in the API request body. 
spec:
  notification_template:
    title: "[{{.Severity}}] {{.Labels.service}} threshold exceeded"
    description: "{{.Labels.env}} crossed {{.ThresholdOp}} {{.Threshold}}"

​
Labels

Labels are user-defined key-value pairs associated with a monitor. Labels identify, categorize, and route alerting time series to specific notifiers through notification policies. Labels don’t affect monitor performance.

​
Annotations

Annotations are notes you can add to a monitor, such as runbooks, links to related dashboards, data links to related traces, and documentation links. Annotations help on-call engineers troubleshoot by surfacing links to related resources or displaying a summary that encapsulates the meaning of the overall query without having to scan the entire Series Legend. Annotation keys are validated against the same regular expression as Prometheus metric names and labels. Annotation values are arbitrary strings. For example, you can create a summary for a monitor that uses labels from the monitor query. Consider the following monitor query that matches on labels for kube_pod_status: 
kube_pod_status{cluster="staging",namespace="test",pod=~"staging-[0-9a-z]+-[0-9a-z]+"}
You can create an annotation that uses variables to substitute values from this monitor query. For example: 
Cluster: {{$labels.cluster}} | Namespace: {{$labels.namespace}} | Pod: {{$labels.pod}} is stale after 10m
When you save your monitor, the annotation replaces variables with values from the monitor query and renders as: 
Cluster: staging | Namespace: test | Pod: staging-37s799fcsj-ekg12 is stale after 10m
You can also create annotations for monitors that link to trace metrics, which lets you provide direct links for your on-call engineers to help diagnose issues. For more information, refer to linking to tracing data. Annotation values support Markdown formatting, including bold text (**text**), inline code (`code`), named links ([label](url)), and plain HTTP URLs. Annotation values can also span multiple lines. Newlines separate paragraphs in the rendered output.

​
Notifiers

Notifiers can reference monitor labels by using the {{ .CommonLabels.LABEL }} variable, replacing LABEL with the monitor label name. Notifiers can reference monitor annotations by using the {{.CommonAnnotations.<ANNOTATION> }} variable, replacing <ANNOTATION> with the monitor annotation name. Building on the previous example, you can create a Slack notifier that references the summary annotation: 
api_version: v1/config
kind: Notifier
spec:
  name: slack-summary-notifier
  slug: slack-notifier
  slack:
    ...
    text: "Summary: {{ .CommonAnnotations.summary }}"
    ...

​
Query

A Prometheus or Graphite query, and a check interval of 15, 30, or 60 seconds that defines how often to check the latest data against configured conditions. When querying monitors, recent data is typically the most relevant. For this reason, Chronosphere recommends the topk function instead of the head_max function. The topk function takes the top K data points at query runtime and returns a filtered list of data points. The head_max function takes the maximum values for each time series across the entire query time range, selects the top K time series based on the maximum value, and then returns the filtered list of time series.

​
Signals

Signals are an optional group of notifications that use a unique combination of labels. When you create a monitor and enable the signals option for notifications, you can add the labels from the monitor’s time series to the group. The following query parameters are available for signals:
ParameterConfigurable?Default
group_waitNo10 seconds
group_intervalNo5 minutes
repeat_intervalYes3600 (seconds)

​
Conditions

Checks run against every time series resulting from the query. You define a condition and sustain period and can assign the resulting alert a severity (warning or critical). If a time series triggers that condition for the sustain period, Observability Platform generates an alert. Evaluating a sustain period of greater than 0 seconds requires at least two checks, defined in the query check interval, before triggering an alert. For example, if the sustain is five seconds, but the check interval is 30 seconds, the alert isn’t triggered for at least 30 seconds, the time it takes for two checks to occur. A sustain period of zero seconds causes the alert to trigger after only one check, but might lead to noisy alerts.

​
Resolve threshold

You can set a resolve threshold on a condition so an alert clears at a different value than where it triggers. Resolve thresholds reduce flapping when a metric repeatedly triggers and resolves while crossing a single threshold, instead of staying in a steady passing or alerting range. If you omit a resolve threshold, the alert resolves using the same threshold and comparison as the trigger. Resolve thresholds are valid only when Alert when value uses one of these comparisons: greater than >, greater than or equal to >=, less than <, or less than or equal to <=. In Chronoctl YAML, Terraform, and the API, those same comparisons are often written as GT (greater than), GEQ (greater than or equal to), LT (less than), and LEQ (less than or equal to). The value of the resolve threshold must be opposite the value set for the alert:
  • For greater than > or greater than or equal to >=, the resolve threshold must be less than or equal to the trigger threshold. For example, trigger when the value is greater than 10 and set the resolve threshold to 5 so the alert clears when the value is at most 5.
  • For less than < or less than or equal to <=, the resolve threshold must be greater than or equal to the trigger threshold. For example, trigger when the value is less than 10 and set the resolve threshold to 10 so the alert clears when the value is at least 10.
A resolve threshold is separate from the resolve window, which is a time-based setting for how long the query must stay clear before the alert resolves. How you configure resolve_value depends on how you create the monitor:
  • Observability Platform: In the Conditions section, set the optional resolve threshold for each numeric comparison (see Create a monitor).
  • Terraform: Optional resolve_value block (enabled and value) inside each condition in series_conditions.
  • Chronoctl or the CreateMonitor API: Optional resolve_value object with enabled and value on each condition. See the API definition for the condition schema.

​
Condition overrides

Defines overrides for specific conditions regarding the selected monitor. Use the EXACT_MATCHER_TYPE matcher to override exact key/value pairs, or the REGEXP_MATCHER_TYPE matcher to match with regular expressions. These are the only supported matcher types for condition overrides.

​
Resolve window

When you have an alert that’s repeatedly resolving and continuing to trigger, define a resolve window. Within the resolve window, Observability Platform suppresses additional triggers for the same alert until the alert hasn’t triggered for the defined duration.
  • Alerts configured with the sustain field trigger if the PromQL expression for the alert returns a populated value for the defined duration.
  • A triggered alert with a defined resolve window resolves after the PromQL expression returns an empty value for the defined duration.
Use resolve windows with conditions or condition overrides. The resolve window defaults to 0s. How you define the resolve window differs depending on how you create the monitor:
  • Terraform: Define the resolve window with the resolve_sustain field in the series_conditions object.
  • Chronoctl or the CreateMonitor API endpoint: Define the resolve window with the resolve_sustain_secs field in the series_conditions object.
  • Observability Platform: In the Conditions pane, define the resolve window with the Resolve when clear for field.
The duration defined in the resolve window definition behaves like the Prometheus keep_firing_for field.
Changing a monitor’s configuration might result in resolve windows not being respected. This behavior can cause an alert to be triggered again, which would enter a new resolve window.

​
Schedule

By default, Observability Platform monitors are constantly active. Limit a monitor’s activity to specific time ranges within a weekly schedule. A schedule defines weekly time ranges when Observability Platform runs the monitor’s queries and evaluates its conditions. The monitor is inactive outside those ranges. Define a schedule when you create or edit a monitor.
Avoid ending a schedule’s time range at 23:59. Using this value can inadvertently stop a monitor from executing during the last minute of the day. To run the monitor through the end of the day, use 24:00. To span days, end the first day at 24:00 and begin the next day at 00:00.
A monitor that’s actively sending alerts resolves after entering an inactive period in a defined schedule. When a monitor re-enters an active state, the monitor must meet defined conditions to trigger a new alert.