> ## Documentation Index > Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt > Use this file to discover all available pages before exploring further. # Use monitors to generate alerts and notifications export const ServiceIcon = props => { return ; }; export const CollectionIcon = props => { const drawPath = "M21 6.5c-1.66 0-3 1.34-3 3 0 .07 0 .14.01.21l-2.03.68c-.64-1.21-1.82-2.09-3.22-2.32V5.91C14.04 5.57 15 4.4 15 3c0-1.66-1.34-3-3-3S9 1.34 9 3c0 1.4.96 2.57 2.25 2.91v2.16c-1.4.23-2.58 1.11-3.22 2.32l-2.04-.68C6 9.64 6 9.57 6 9.5c0-1.66-1.34-3-3-3s-3 1.34-3 3 1.34 3 3 3c1.06 0 1.98-.55 2.52-1.37l2.03.68c-.2 1.29.17 2.66 1.09 3.69l-1.41 1.77C6.85 17.09 6.44 17 6 17c-1.66 0-3 1.34-3 3s1.34 3 3 3 3-1.34 3-3c0-.68-.22-1.3-.6-1.8l1.41-1.77c1.36.76 3.02.75 4.37 0l1.41 1.77c-.37.5-.59 1.12-.59 1.8 0 1.66 1.34 3 3 3s3-1.34 3-3-1.34-3-3-3c-.44 0-.85.09-1.23.26l-1.41-1.77c.93-1.04 1.29-2.4 1.09-3.69l2.03-.68c.53.82 1.46 1.37 2.52 1.37 1.66 0 3-1.34 3-3S22.66 6.5 21 6.5zm-18 4c-.55 0-1-.45-1-1s.45-1 1-1 1 .45 1 1-.45 1-1 1zM6 21c-.55 0-1-.45-1-1s.45-1 1-1 1 .45 1 1-.45 1-1 1zm5-18c0-.55.45-1 1-1s1 .45 1 1-.45 1-1 1-1-.45-1-1zm1 12c-1.38 0-2.5-1.12-2.5-2.5S10.62 10 12 10s2.5 1.12 2.5 2.5S13.38 15 12 15zm6 4c.55 0 1 .45 1 1s-.45 1-1 1-1-.45-1-1 .45-1 1-1zm3-8.5c-.55 0-1-.45-1-1s.45-1 1-1 1 .45 1 1-.45 1-1 1z"; return ; }; One of the reasons to ingest and store time series data is to know when data meets or doesn't meet certain criteria. Use Chronosphere Observability Platform alerting to generate alerts and notifications from data, whether it's about your system or about your usage of Observability Platform itself. Compare your monitor configurations to historical data to ensure your thresholds meet your needs. Observability Platform lets you designate certain monitors as your [favorites](/navigate/favorites), listing them on your personal home page and prioritizing them in global search results. ## View available monitors Select from the following methods to view and filter monitors. To query and get detailed information about monitors, see [monitor actions](/investigate/alerts/monitors/monitor-actions). To display a list of defined monitors, in the navigation menu, select ** Alerting > Monitors**. The list of monitors includes an **Alert state** column. Each monitor displays a badge showing the icon for its most severe active state, with a text label listing all active alert counts. For example, a monitor with two active critical alerts and one active warning alert shows the critical icon with the label **2 Critical, 1 Warning**. | Icon | State | Description | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | ---------------------------------------------------------------------- | | | **Critical** | Monitor has at least one active critical alert. | | | **Warning** | Monitor has at least one active warning alert, and no critical alerts. | | Muted bell icon | **Muted** | Monitor's alerts are muted. | | | **Passing** | Monitor has no active alerts. | Use the following methods to filter your monitors: * Using the **Search monitors** search box (an **OR** filter). * By team, using the **Select a team** dropdown. * By owner, using the **Select an owner** dropdown. The icon indicates the monitor is part of a [collection](/administer/collections). The icon indicates this monitor is part of a service. * By notification policy, using the **Select a notification policy** dropdown. * By error status: * **All**: Default, displays all monitors. * **Alerting**: Monitors currently in alert status. * **Critical**: Monitors in a critical alert status. * **Muted**: Displays only muted monitors. * To filter the table to display only your favorite monitors, enable the **View only my favorites** toggle. * **Include connected monitors**: If you filter monitors by owner with the toggle disabled, only monitors owned by that owner are returned. When the toggle is enabled, your filter includes monitors that are connected to that owner, even if they aren't owned by that owner. Connections are based on [collections](/administer/collections/home). Monitors with defined signals display the file tree icon. To view the signals from a displayed monitor, click the name of the monitor from the list. From a monitor's detail page, click the name of a signal from the **Signals** section to filter the query results to alerts only from that signal. To search for a specific monitor: 1. Click the search bar to focus on it, or use the keyboard shortcut `Control+K` (`Command+K` on macOS). 2. Begin typing any part of the monitor's name. 3. Optional: Click the filters for all other listed resource types at the top of the search results to remove them and display only monitors. 4. Click the search result you're interested in, or use the arrow keys to select it and press enter, to go to that monitor. To use [Chronoctl](/tooling/chronoctl) to return all monitors, use the `chronoctl monitors list` command: ```shell theme={null} chronoctl monitors list ``` To filter for a specific monitor, add the `slugs` argument to the command: ```shell /SLUG/ theme={null} chronoctl monitors list --slugs SLUG ``` Replace *`SLUG`* with the slug for the monitor you want to display. [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) in Observability Platform to view the monitor's Chronoctl YAML representation. [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) in Observability Platform to view a monitor's Terraform representation. To complete this action with the Chronosphere API, use the [`ListMonitors`](/tooling/api-info/definition/operations/ListMonitors) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. For information about what's available on an individual monitor's detail page, including query results, series legend, active alerts, alert history, and change events, see [Monitor details](/investigate/alerts/monitors/monitor-details). ## Create a monitor Most monitors alert when a value matches a specific condition, such as when an error condition defined by the query lasts longer than one minute. You can also choose to alert when a value doesn't exist, such as when a host stops sending metrics and is likely unavailable. The **Web** procedure in [Create monitors](#create-monitors) walks through missing data using **not exists** and **signal not exists** in **Conditions**. For how those choices map to configuration, signal grouping, and Prometheus, logs, or Graphite support, see [Missing data conditions](#missing-data-conditions). To receive alerts when a host stops sending metrics, create a separate monitor for each host and scope the monitor query to that host. ### Prerequisites Before creating a monitor, complete the following tasks: 1. Create a [notifier](/investigate/alerts/notifications/notifiers) to define where to deliver alerts and who to notify. 2. Create a [notification policy](/investigate/alerts/notifications/policies) to determine how to route notifications to notifiers based on signals that trigger from your monitor. You select the notifier you created for the critical or warning conditions on the notification policy. ### Create monitors After completing the [prerequisite tasks](#prerequisites), use any of the following methods to create a new monitor. When creating or editing a monitor in Observability Platform, you can simulate and test alerts to see how an alert would have performed against historical data. Use backtesting to review how your alert would have performed if it had been defined in the past. Chronosphere recommends a minimum query interval of 15 seconds. There can be a 10-second delay between an alert trigger and the notifier activation. Use one of the following procedures to create a monitor. To add a new monitor: 1. In the navigation menu, select one of these locations: * ** Alerting > Monitors**. * **Platform > Collections**, and then select the collection you want to create a monitor for. This can be a standard collection or a [service](/observe/services). 2. Create the monitor: * From the **Monitors** page, click **Create monitor** to open the **Add Monitor** panel. * To create a monitor by duplicating an existing monitor, click the monitor on the **Monitors** page, then click ** Duplicate** to open the **Duplicate monitor** panel. * From the **Collections** page, in the **Monitors** panel, click **Manage**, then click **New monitor**. 3. Enter the information for the monitor based on its [data model](/investigate/alerts/monitors/data-model). 4. Select an **Owner** to organize and filter your monitor. You can select a collection or a service. 5. Enter a **Monitor Name**, which you can change after creating the monitor. Monitor names are static strings and don't accept label variables, such as `$labels.LABEL_NAME`. 6. Choose a **Notification Policy** to determine which notification policy to use at a particular alert severity. 7. Enter **Labels** as key/value pairs to categorize and filter monitors. 8. In the **Query** section, choose the type of query you want to enter: * **Prometheus**: Enter a valid Prometheus query. Click **Edit in Query Builder** to open your query in the [Query Builder](/investigate/querying/metrics/builder), where you can construct, optimize, and debug your query before saving it. After modifying your query, click **Done** to return to the **Add Monitor** page. * **Graphite**: Enter a valid Graphite query. * **Logs**: Enter a valid log query, which must include the [`make-series`](/investigate/querying/query-logs/query-syntax#make-series) operator with a specified `step` size to return data. This operator uses the [`count()`](/investigate/querying/query-logs/query-syntax#count) function by default, but you can specify different operators instead. For example, the following query creates a time chart that includes the average for `latencyInSeconds`. The `step` parameter defines the time step for each bucket in [Prometheus time duration format](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-durations): ```text theme={null} severity = "WARNING" | make-series avg(latencyInSeconds) step 15m by severity, service ``` If the log query includes a field that contains a period in its name and you want to use signals to group notifications, use an [alias for that field name](/investigate/querying/query-logs/query-syntax#alias-field-names). Otherwise, periods are converted to underscores in the generated visualization. 9. Use these options to validate and update your query: * Click **Check Query** to validate your query and preview query results. In the query preview, use the following options to understand your query: * Toggle **Show thresholds** to display the monitor's defined thresholds. * Select a time range up to the present in the ** [time range selector](/navigate/time-ranges)**. If your selected time period has too many alerts, or the entire graph appears to display in alerted status, reduce the selected time period. If multiple alerts would have triggered simultaneously, only one threshold marker displays. The banner shows the correct number of alerts. For example, if a critical and a warning would trigger at the same time, only one alert displays on the graph. The banner shows two alerts would have triggered. * Click **Open in Explorer** to open your query in [Metrics Explorer](/investigate/querying/metrics/explorer), where you can review your query for syntax errors and make necessary changes. 10. For Prometheus queries, test monitor conditions by reviewing when a monitor would have triggered, based on historical data. The preview reflects existing monitor schedules, signal grouping, and overrides: * Use the **Show alert durations** toggle to display the time period over which the alert would have been active. * Toggle **Simulate alerts** to backtest your condition against existing data. You must define at least one condition for alert simulations to work. Alert simulations use existing data, and can't predict future alerts. If your selected query returns too much data, the graph displays an error. Chronosphere recommends selecting shorter time periods for testing, when possible. Alert simulation isn't available outside the [raw data retention period](/administer/limits-licensing/licensing#ingestion-limits-and-retention-policies). 11. Optional: Group alerts based on the results returned from the query by choosing an option in the **Signals** section. [Signals](/investigate/alerts/notifications/signals) use a unique set of labels to create groups of notifications when a monitor alert triggers or resolves. If you select **per signal (multiple alerts)** to generate multiple alerts, enter a label key that differs in name and casing from the label you enter in the **Key** field in the **Labels** section. For example, if you enter `environment` in the **Key** field, you might use `Environments` as the **Label Key** to match on. [Pinned scopes](/navigate/pinned-scopes) can be used as a **Label Key**. 12. Define a condition and sustain period in the **Conditions** section, and assign the resulting alert a severity (warning or critical). In the **Sustain** field, enter a value followed by an abbreviated unit such as `60s`. Valid units are `s` (seconds), `m` (minutes), `h` (hours), or `d` (days). The dialog also displays the notifiers associated with the monitor for reference. To generate an alert when the entire monitor query returns no results, select **not exists** in the **Alert when value** dropdown. Select **signal not exists** when alerts should respect [signal grouping](/investigate/alerts/notifications/signals). For example, all series in a signal must be missing, or any single series when using per-series signals. The **signal not exists** option requires a sustain duration between 5 minutes and 24 hours. 13. The fields for defining resolution time depend on the comparison selector you choose. For **Resolve when clear for**, enter a time period for the resolve window in the **Resolve** field as a value followed by an abbreviated unit such as `30s`. Valid units are `s` (seconds), `m` (minutes), `h` (hours), or `d` (days). When **Alert when value** uses one of these comparisons: greater than `>`, greater than or equal to `>=`, less than `<`, or less than or equal to `<=`, set a **Resolve threshold**. 1. Click **Add threshold**. 2. Enter a **Resolve value**. The alert doesn't clear until the series crosses that separate boundary. If you omit this section, resolution uses the same threshold as the trigger. See [Resolve threshold](/investigate/alerts/monitors/data-model#resolve-threshold). 14. In the **Monitor schedule** section, choose when Observability Platform evaluates the monitor and can send alerts: * **Always on**: The monitor is evaluated continuously and can alert whenever conditions are met. This is the default behavior when you don't restrict evaluation windows. * **Scheduled**: The monitor is evaluated only during the time windows you define. 1. Select **Time zone** for the schedule. 2. For each time window, set **Start** and **End** using 24-hour times in `HH:MM` form. 3. Under **Repeat every**, select the days of the week that window applies to. 4. Use **Add range** to add more than one weekly window. * **Disabled**: The monitor doesn't send alerts. Use this mode to keep the monitor definition without active alerting. For schedule behavior, alerting, resolution, and end times, for example `24:00` instead of `23:59`, see [Schedule](/investigate/alerts/monitors/data-model#schedule) in the monitor data model. 15. Add notes for the monitor in the **Annotations** section, such as runbook summaries and descriptions. 16. Optional: In the **Links** section, add [templated links](/investigate/alerts/monitors/annotation-links) to dashboards, Logs Explorer, and external runbooks. 17. Optional: To customize the notification title and description, configure a [notification template](/investigate/alerts/monitors/notification-templates). 18. Click **Save**. To create a monitor with [Chronoctl](/tooling/chronoctl): 1. Run the following command to generate a sample monitor configuration you can use as a template: ```shell theme={null} chronoctl monitors scaffold ``` In the template, `kind: Monitor` defines an individual monitor. 2. With a completed definition, submit it with: ```shell /FILE_NAME/ theme={null} chronoctl monitors create -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. See the [Chronoctl examples](#chronoctl-example-prometheus) for a completed monitor definition. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. To create a monitor with [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file and add the definition by using the `chronosphere_monitor` type, followed by a name in a resource declaration. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` See the [Terraform examples](#terraform-example-prometheus) for a completed monitor resource. To complete this action with the Chronosphere API, use the [`CreateMonitor`](/tooling/api-info/definition/operations/CreateMonitor) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. #### Missing data conditions A monitor can have different conditions that indicate missing data. See the [CreateMonitor](/tooling/api-info/definition/operations/CreateMonitor) endpoint for more information. A `NOT_EXISTS` condition triggers only when the entire monitor query returns no time series. Use a `SIGNAL_NOT_EXISTS` condition when a missing data alert should use the same [signal grouping](/investigate/alerts/notifications/signals) as the rest of the monitor. `SIGNAL_NOT_EXISTS` triggers when a specific signal no longer has matching series (for example after series for one label value drop out while others remain). That behavior differs from `NOT_EXISTS`, which only considers whether the full query result is empty. Monitors can have the following signals to group time series: * Per monitor (no `signal_grouping`, or equivalent): same as `NOT_EXISTS`. The alert triggers when every series from the query is missing. * Per signal (`label_names`): the alert triggers when every series in a signal is missing (all series that share that signal's label set). * Per series (`signal_per_series`): the alert triggers when any individual series disappears. For example, a **Prometheus** monitor runs `sum by (cluster) (error_rate())` with signal grouping on `cluster`, so each cluster is its own signal. 1. While `cluster="dev"` and `cluster="prod"` both return series, neither operator triggers a missing data alert from that condition alone. 2. If series for `prod` disappear but `dev` still returns data, `SIGNAL_NOT_EXISTS` can alert for the `prod` signal. `NOT_EXISTS` doesn't trigger, because the query still returns results. 3. If `dev` then disappears, the query returns no time series. `SIGNAL_NOT_EXISTS` can alert for the `dev` signal as well, and `NOT_EXISTS` triggers because the entire query is empty. `SIGNAL_NOT_EXISTS` alerts can resolve automatically after a period once the missing data condition clears. `SIGNAL_NOT_EXISTS` works for **Prometheus** monitors, not for **Graphite** queries. The **Sustain** field is required and must be between 5 minutes and 24 hours. Omit `value` (or set it to zero), the same as for `EXISTS` and `NOT_EXISTS`. Use `SIGNAL_NOT_EXISTS` only on default conditions, not in [condition overrides](#override-a-monitor-alert). Resolve thresholds (`resolve_value`) aren't supported for this operator. In Chronoctl YAML and Terraform, set `op` to `NOT_EXISTS` or `SIGNAL_NOT_EXISTS` on a condition under `series_conditions`, with the sustain fields your format expects and with `value` omitted or zero. The [Chronoctl](#chronoctl-example-prometheus) and [Terraform](#terraform-example-prometheus) examples show the full monitor definition. In the Chronoctl YAML, use the inline comments before the sample threshold conditions when replacing or adding a missing data condition. In Terraform, use the same `series_conditions` layout and the comments on `op` and `value` in the example resource as a guide. The following examples show complete monitor definitions for each tool and query type. Each example groups series into signals based on the `source` and `service_environment` label keys, and includes a `schedule` that runs the monitor on Mondays from 7:00 to 10:10 and 15:00 to 22:30, and Thursdays from 21:15 through the end of the day. All times are in UTC. If you define `label_names` in the `signal_grouping` section, enter a label name that differs in name and casing from the label you enter in the `labels` section. For example, if you enter `environment` as a key in the `labels` section, you might use `Environments` in the `label_names` section. The following YAML defines a Prometheus monitor named `Disk Getting Full`. The `series_conditions` trigger a warning notification when the value exceeds 30 for more than 300 seconds, and a critical notification when it exceeds 60 for more than 300 seconds. ```yaml theme={null} api_version: v1/config kind: Monitor spec: # Required name of the monitor. Can be modified after the monitor is created. name: Disk Getting Full # PromQL query. If set, you can't set graphite_query. prometheus_query: max(disk:last{measurement="used_percent"}) by (source, service_environment, region) # Annotations are visible in notifications generated by this monitor. # You can template annotations with labels from notifications. annotations: key_1: "{{ $labels.job }}" # Slug of the collection the monitor belongs to. collection_slug: loadgen # Optional setting for configuring how often alerts are evaluated. # Defaults to 60 seconds. interval_secs: 60 # Labels are visible in notifications generated by this monitor, # and can be used to route alerts with notification overrides. labels: key_1: kubernetes_cluster # Optional notification policy used to route alerts generated by the monitor. notification_policy_slug: custom-notification-policy schedule: # The timezone of the time ranges. timezone: UTC weekly_schedule: monday: active: ONLY_DURING_RANGES # The time ranges that the monitor is active on this day. Required if # active is set to ONLY_DURING_RANGES. ranges: - # End time in the in format ":", such as "15:30". end_hh_mm: "15:00" # Start time in the in format ":", such as "15:30". start_hh_mm: "10:10" tuesday: active: NEVER wednesday: active: NEVER thursday: active: ONLY_DURING_RANGES # The time ranges that the monitor is active on this day. Required if ranges: # active is set to ONLY_DURING_RANGES. - # End time in the in format ":", such as "15:30". end_hh_mm: "24:00" # Start time in the in format ":", such as "15:30". start_hh_mm: "21:15" friday: active: NEVER saturday: active: NEVER sunday: active: NEVER # Conditions evaluated against each queried series to determine the severity of each series. series_conditions: defaults: critical: # List of conditions to evaluate against a series. # Only one condition must match to assign a severity to a signal. conditions: # Missing data: `NOT_EXISTS` (entire query empty) or `SIGNAL_NOT_EXISTS` # (signal-aware; sustain must be from 5m to 24h; not for Graphite). - op: GT # How long the op operation needs to evaluate for the condition # to evaluate to true. sustain_secs: 300 # The value to compare to the metric value using the op operation. value: 60 # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). # resolve_value: # enabled: true # value: 50 # How long the operation needs to evaluate false to resolve resolve_sustain_secs: 60 warn: # List of conditions to evaluate against a series. # Only one condition must match to assign a severity to a signal. conditions: - op: GT # How long the op operation needs to evaluate for the condition # to evaluate to true. sustain_secs: 300 # The value to compare to the metric value using the op operation. value: 30 # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). # resolve_value: # enabled: true # value: 20 # How long the operation needs to evaluate false to resolve resolve_sustain_secs: 60 # Defines how the set of series from the query are split into signals. signal_grouping: label_names: - source - service_environment # If true, each series will have its own signal and label_names can't be set. signal_per_series: false ``` The following YAML defines a logs monitor named `Kubernetes errors in production us-west`. The `series_conditions` trigger a warning notification when the error count exceeds 30 for more than 300 seconds, and a critical notification when it exceeds 60 for more than 300 seconds. ```yaml theme={null} api_version: v1/config kind: Monitor spec: # Required name of the monitor. Can be modified after the monitor is created. name: Kubernetes errors in production us-west # Logging query to return data for. logging_query: severity = "ERROR" AND kubernetes.cluster_name = "production-us-west" | make-series by service # Annotations are visible in notifications generated by this monitor. # You can template annotations with labels from notifications. annotations: key_1: "{{ $labels.job }}" # Slug of the collection the monitor belongs to. collection_slug: production-team # Optional setting for configuring how often alerts are evaluated. # Defaults to 60 seconds. interval_secs: 60 # Labels are visible in notifications generated by this monitor, # and can be used to route alerts with notification overrides. labels: key_1: kubernetes_cluster # Optional notification policy used to route alerts generated by the monitor. notification_policy_slug: custom-notification-policy schedule: # The timezone of the time ranges. timezone: UTC weekly_schedule: monday: active: ONLY_DURING_RANGES # The time ranges that the monitor is active on this day. Required if # active is set to ONLY_DURING_RANGES. ranges: - # End time in the in format ":", such as "15:30". end_hh_mm: "15:00" # Start time in the in format ":", such as "15:30". start_hh_mm: "10:10" tuesday: active: NEVER wednesday: active: NEVER thursday: active: ONLY_DURING_RANGES # The time ranges that the monitor is active on this day. Required if ranges: # active is set to ONLY_DURING_RANGES. - # End time in the in format ":", such as "15:30". end_hh_mm: "24:00" # Start time in the in format ":", such as "15:30". start_hh_mm: "21:15" friday: active: NEVER saturday: active: NEVER sunday: active: NEVER # Conditions evaluated against each queried series to determine the severity of each series. series_conditions: defaults: critical: # List of conditions to evaluate against a series. # Only one condition must match to assign a severity to a signal. conditions: # Missing data: `NOT_EXISTS` (entire query empty) or `SIGNAL_NOT_EXISTS` # (signal-aware; sustain must be from 5m to 24h; not for Graphite). - op: GT # How long the op operation needs to evaluate for the condition # to evaluate to true. sustain_secs: 300 # The value to compare to the metric value using the op operation. value: 60 # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). # resolve_value: # enabled: true # value: 50 # How long the operation needs to evaluate false to resolve resolve_sustain_secs: 60 warn: # List of conditions to evaluate against a series. # Only one condition must match to assign a severity to a signal. conditions: - op: GT # How long the op operation needs to evaluate for the condition # to evaluate to true. sustain_secs: 300 # The value to compare to the metric value using the op operation. value: 30 # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). # resolve_value: # enabled: true # value: 20 # How long the operation needs to evaluate false to resolve resolve_sustain_secs: 60 # Defines how the set of series from the query are split into signals. signal_grouping: label_names: - source - service_environment # If true, each series will have its own signal and label_names can't be set. signal_per_series: false ``` The following Terraform resource creates a Prometheus monitor that Terraform refers to by `infra`, with a human-readable name of `Infra Example monitor`. ```terraform theme={null} resource "chronosphere_monitor" "infra" { name = "Infra Example monitor" # Reference to the collection the alert belongs to. collection_id = chronosphere_collection.infra.id # Override the notification policy. # By default, uses the policy from the collection_id. notification_policy_id = chronosphere_collection.infra_testing.id # Arbitrary set of labels to assign to the alert. labels = { "priority" = "sev-1" } # Arbitrary set of annotations to include in alert notifications. annotations = { "runbook" = "http://default-runbook" } # Interval at which to evaluate the monitor, for example 15s, 30s, or 60s. # Defaults to 60s. interval = "30s" query { # PromQL query to evaluate for the alert. # Alternatively, you can use graphite_expr instead. prometheus_expr = "sum (rate(grpc_server_handled_total{grpc_code!="OK"}[1m])) by (app, grpc_service, grpc_method)" } # The remaining examples are optional signals specifying how to group the # series returned from the query. # No signal_grouping clause = Per monitor # signal_grouping with label_names set = Per signal for labels set # signal_grouping with signal_per_series set to true = Per series signal_grouping { # Set of labels names used to split series into signals. # Each unique combination of labels results in its own signal. label_names = ["app", "grpc_service"] # As an alternative to label_names, signal_per_series creates an alert for # every resulting series from the query. # signal_per_series = true } # Container for the conditions determining the severity of each series from the query. # The highest severity series of a signal determines that signal's severity. series_conditions { # Condition assigning a warn threshold for series above a certain threshold. condition { # Severity of the condition, which can be "warn" or "critical". severity = "warn" # Value to compare against each series from the query result. # For EXISTS, NOT_EXISTS, or SIGNAL_NOT_EXISTS, value must be zero or omitted. value = 5.0 # Operator to use when comparing the query result versus the threshold. # Valid values include GT, LT, LEQ, GEQ, EQ, NEQ, EXISTS, NOT_EXISTS, # SIGNAL_NOT_EXISTS. op = "GT" # Amount of time the query needs to fail the condition check before # an alert is triggered. Must be an integer. Accepts one of s (seconds), m # (minutes), or h (hours) as units. Optional. sustain = "240s" # Amount of time the query needs to no longer trigger before resolving. Must be # an integer. Accepts one of s (seconds), m (minutes), or h (hours) as units. resolve_sustain = "60s" } condition { severity = "critical" value = 10.0 op = "GT" sustain = "120s" resolve_sustain = "60s" # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). resolve_value { enabled = true value = 5.0 } } # Multiple optional overrides can be defined for different sets of conditions # to series with matching labels. override { # One or more matchers for labels on a series. label_matcher { # Name of the label name = "app" # How to match the label, which can be "EXACT_MATCHER_TYPE" or # "REGEXP_MATCHER_TYPE". type = "EXACT_MATCHER_TYPE" # Value of the label. value = "dbmon" } condition { severity = "critical" value = 1.0 op = "GT" sustain = "60s" } } } # If you define a schedule, Observability Platform evaluates the monitor only during # the specified time ranges. The monitor is inactive during all unspecified # time ranges. # If you define an empty schedule, Observability Platform never evaluates the monitor. schedule { # Valid values: Any IANA timezone string timezone = "UTC" range { # Time range for the monitor schedule. Valid values for day can be full # day names, such as "Sunday" or "Monday". # Valid time values must be specified in the range of 00:00 to 24:00. day = "Monday" start = "07:00" end = "10:10" } range { day = "Monday" start = "15:00" end = "22:30" } range { day = "Thursday" start = "21:15" end = "24:00" } } } ``` The following Terraform resource creates a logs monitor that Terraform refers to by `k8s_production`, with a human-readable name of `Kubernetes errors in production us-west`. ```terraform theme={null} resource "chronosphere_monitor" "k8s_production" { name = "Kubernetes errors in production us-west" # Reference to the collection the alert belongs to. collection_id = chronosphere_collection.k8s_production.id # Override the notification policy. # By default, uses the policy from the collection_id. notification_policy_id = chronosphere_collection.k8s_testing.id # Arbitrary set of labels to assign to the alert. labels = { "priority" = "sev-1" } # Arbitrary set of annotations to include in alert notifications. annotations = { "runbook" = "http://default-runbook" } # Interval at which to evaluate the monitor, for example 15s, 30s, or 60s. # Defaults to 60s. interval = "30s" query { # Logging query to evaluate. logging_query = "severity='ERROR' AND kubernetes.cluster_name='production-us-west' | make-series by service" } # The remaining examples are optional signals specifying how to group the # series returned from the query. # No signal_grouping clause = Per monitor # signal_grouping with label_names set = Per signal for labels set # signal_grouping with signal_per_series set to true = Per series signal_grouping { # Set of labels names used to split series into signals. # Each unique combination of labels results in its own signal. label_names = ["app", "grpc_service"] # As an alternative to label_names, signal_per_series creates an alert for # every resulting series from the query. # signal_per_series = true } # Container for the conditions determining the severity of each series from the query. # The highest severity series of a signal determines that signal's severity. series_conditions { # Condition assigning a warn threshold for series above a certain threshold. condition { # Severity of the condition, which can be "warn" or "critical". severity = "warn" # Value to compare against each series from the query result. # For EXISTS, NOT_EXISTS, or SIGNAL_NOT_EXISTS, value must be zero or omitted. value = 5.0 # Operator to use when comparing the query result versus the threshold. # Valid values include GT, LT, LEQ, GEQ, EQ, NEQ, EXISTS, NOT_EXISTS, # SIGNAL_NOT_EXISTS. op = "GT" # Amount of time the query needs to fail the condition check before # an alert is triggered. Must be an integer. Accepts one of s (seconds), m # (minutes), or h (hours) as units. Optional. sustain = "240s" # Amount of time the query needs to no longer trigger before resolving. Must be # an integer. Accepts one of s (seconds), m (minutes), or h (hours) as units. resolve_sustain = "60s" } condition { severity = "critical" value = 10.0 op = "GT" sustain = "120s" resolve_sustain = "60s" # Optional. resolve at a different threshold (GT/GEQ/LT/LEQ only). resolve_value { enabled = true value = 5.0 } } # Multiple optional overrides can be defined for different sets of conditions # to series with matching labels. override { # One or more matchers for labels on a series. label_matcher { # Name of the label name = "app" # How to match the label, which can be "EXACT_MATCHER_TYPE" or # "REGEXP_MATCHER_TYPE". type = "EXACT_MATCHER_TYPE" # Value of the label. value = "dbmon" } condition { severity = "critical" value = 1.0 op = "GT" sustain = "60s" } } } # If you define a schedule, Observability Platform evaluates the monitor only during # the specified time ranges. The monitor is inactive during all unspecified # time ranges. # If you define an empty schedule, Observability Platform never evaluates the monitor. schedule { # Valid values: Any IANA timezone string timezone = "UTC" range { # Time range for the monitor schedule. Valid values for day can be full # day names, such as "Sunday" or "Monday". # Valid time values must be specified in the range of 00:00 to 24:00. day = "Monday" start = "07:00" end = "10:10" } range { day = "Monday" start = "15:00" end = "22:30" } range { day = "Thursday" start = "21:15" end = "24:00" } } } ``` ## Edit a monitor Select from the following methods to edit monitors. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To edit a monitor: When editing a monitor, use the same interface as when you created it. 1. In the navigation menu, select ** Alerting > Monitors**. 2. Click the name of the monitor you want to edit. 3. On the monitor's page, click ** Edit** in the page header. The **Edit monitor** drawer opens with the same fields as when you create a monitor. 4. Make your edits, and then click **Save**. Refer to the [monitor data model](/investigate/alerts/monitors/data-model) for specific definitions. To edit a monitor using [Chronoctl](/tooling/chronoctl): 1. [View the monitor's Chronoctl YAML](#view-available-monitors). 2. Modify its properties and apply the changes with the same process as [creating a monitor](#create-a-monitor). Chronoctl updates the monitor's properties if the definition has the same slug. You can also use the following process if you already have a definition file: 1. Update the monitor's definition file. 2. Run the following command to submit the changes: ```shell /FILE_NAME/ theme={null} chronoctl monitors update -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. You can also use the Code Config tool to view the monitor's Chronoctl YAML representation. For details, see [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool). To edit a monitor using [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file that updates the resource's existing properties. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` You can also use the Code Config tool to view the monitor's Terraform representation. For details, see [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool). To complete this action with the Chronosphere API, use the [`UpdateMonitor`](/tooling/api-info/definition/operations/UpdateMonitor) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Override a monitor alert You can override the default conditions that define when an alert triggers for a monitor. This override is similar to [overriding a notification policy](/investigate/alerts/notifications/policies#override-notification-policy-defaults) that routes a notification to a notifier other than the specified default. On a monitor, you can specify a condition override to use a separate threshold for certain series. For example, a monitor might have a default threshold of `>100` but you specify an override threshold of `>50` where the label key/value pair is `cluster=production`. You can specify any label as a matcher for a monitor condition override. If no override matches the defined conditions, Observability Platform applies the default conditions. Additionally: * Overrides must specify at least one matcher, and meet every matcher condition to apply the override. * Observability Platform evaluates overrides in the listed order. When an override matches, the remaining overrides and defaults are ignored. * Overrides don't inherit any properties from the default conditions. For example, if the default policy route specifies `warn` and `critical` notifiers but the override specifies only `critical` notifiers, the notifier doesn't send `warn` notifications. * You can't use `NOT_EXISTS` or `SIGNAL_NOT_EXISTS` on override conditions; use them only on default conditions. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To specify a monitor alert override: 1. In the navigation menu, select ** Alerting > Monitors**. 2. Click the name of the monitor you want to specify an override for. 3. On the monitor's page, click ** Edit** in the page header. The **Edit monitor** drawer opens. 4. In the **Condition Override** section, click the plus icon to display the override fields. 5. Select **Exact** or **Regex** as the matcher type, and enter the key/value pair to match on for the override. 6. Select **Critical** or **Warn** as the override severity. 7. Define the match condition and sustain duration. 8. Optional: Set a **Resolve threshold** using the same rules as [default conditions](/investigate/alerts/monitors/data-model#resolve-threshold). 9. Click **Save** to apply the override changes. 1. Use [Terraform](/tooling/infrastructure/terraform) to add an override to the resource's existing properties and apply the changes. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` The following Terraform HCL definition adds monitor overrides that match on specific labels, and associate either a `warn` or `critical` alert based on the matching key/value pairs. ```text theme={null} series_conditions { condition { severity = "critical" op = "GT" value = 20 sustain = "180s" resolve_sustain = "180s" } condition { severity = "warn" op = "GT" value = 10 sustain = "180s" } override { label_matcher { type = "EXACT_MATCHER_TYPE" name = "namespace" value = "production" } label_matcher { type = "EXACT_MATCHER_TYPE" name = "container" value = "k8s_prod" } condition { severity = "critical" op = "GT" value = 50 sustain = "180s" } condition { severity = "warn" op = "GT" value = 40 sustain = "180s" } } override { label_matcher { type = "EXACT_MATCHER_TYPE" name = "container" value = "billing_svc" } condition { severity = "critical" op = "GT" value = 30 } condition { # only one of these conditions needs to be met severity = "critical" op = "GT" value = 25 sustain = "360s" } } } ``` Based on the previous override, the following descriptions explain how incoming series impact alerting conditions for the monitor: * The following series doesn't match any override conditions, so the default alert conditions apply: ```yaml theme={null} {namespace="production", container="gateway", job="gateway-scraper"} 5 ``` * The following series matches the `warn` condition for the first override, so the alert uses a `warn` notifier: ```yaml theme={null} {namespace="production", container="k8s_prod", job="gateway-scraper"} 45 ``` * The following series matches the `critical` condition defined in the override, so the alert uses a `critical` notifier: ```yaml theme={null} {namespace="foo3", container="bar", job="gateway-scraper"} 27 ``` ## Delete a monitor Select from the following methods to delete monitors. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To delete a monitor: 1. In the navigation menu, select ** Alerting > Monitors**. 2. Click the name of the monitor you want to delete. 3. On the monitor's page, click ** Edit** in the page header. The **Edit monitor** drawer opens. 4. In the drawer footer, click **Delete monitor**. 5. In the confirmation dialog, click **Delete**. To delete a monitor with [Chronoctl](/tooling/chronoctl), use the `chronoctl monitors delete` command: ```shell /SLUG/ theme={null} chronoctl monitors delete SLUG ``` Replace *`SLUG`* with the slug of the monitor you want to delete. For example, to delete a monitor with the slug `infra-example-monitor`: ```shell theme={null} chronoctl monitors delete infra-example-monitor ``` To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteMonitor`](/tooling/api-info/definition/operations/DeleteMonitor) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Use annotations with monitors Annotations provide runbook summaries, descriptions, and other text context for on-call engineers diagnosing issues. Use the separate [Links](/investigate/alerts/monitors/annotation-links) section in the monitor editor to add templated URLs to dashboards, Logs Explorer, and external resources. You can also add link-valued annotations through Terraform, Chronoctl, or the API. For cross-telemetry linking patterns, see [Create links to related information](/investigate/querying/create-links). Reference Prometheus Alertmanager variables in annotation values with the `{{ .VARIABLE_NAME }}` syntax. Annotations access monitor labels with `{{ .CommonLabels.LABEL }}` and the alerting metric's labels with `{{ .Labels.LABEL }}`. In both patterns, replace *`LABEL`* with the label name. Labels referenced in Alertmanager variables must be present in the alerting time series. If a label is absent, the variable renders as empty in notifications. For a reference list of alerting variables and templating functions, see the [Alertmanager documentation](https://prometheus.io/docs/alerting/latest/notifications/). Annotation values support Markdown formatting, including bold text (`**text**`), inline code (`` `code` ``), named links (`[label](url)`), and plain HTTP URLs. Annotation values can also span multiple lines using the multiline annotation input. Each paragraph break renders as a new line in the monitor details view. Observability Platform also provides additional variables in annotation values: | Shorthand | Go template | Description | | ----------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------- | | `$value` | `{{ .Value }}` | The numeric query result value at the time the alert triggered. | | `$labels.LABEL` | `{{ .Labels.LABEL }}` | A label from the alerting time series. Replace *`LABEL`* with the label name. | | `$externalLabels.LABEL` | `{{ .ExternalLabels.LABEL }}` | A system-level external label configured in Observability Platform. Replace *`LABEL`* with the label name. | | `$externalURL` | `{{ .ExternalURL }}` | The base URL of your Observability Platform instance. | | `$startTime` | `{{ .StartTime }}` | The time the alert started firing. | | `$endTime` | `{{ .EndTime }}` | The time the alert resolved. | Use shorthand syntax to reference label names that contain dots, such as `label.name`. You can also use the `humanize` template function to format numeric values. For example, `{{ $value | humanize }}` renders `1500000` as `1.5M`. To add annotations to a monitor: 1. [Create a monitor](#create-a-monitor). 2. In the **Annotations** section, add a description for your annotation in the **Key** field, and text or links in the **Value** field. For example, you might add the following key/value pairs as annotations: | Key | Value | | ----------- | ------------------------------------------------------------------------------------------------------------------------- | | summary | Instance `{{ $labels.instance }}` is down | | description | Container `{{ $labels.namespace }}`/`{{ $labels.pod }}`/`{{ $labels.container }}` terminated with `{{ $labels.reason }}`. | | value | Current value: `{{ $value }}` | | runbook | `http://default-runbook` | With Markdown formatting enabled, annotation values support richer formatting. For example, a `description` value might look like: ```text wrap theme={null} **Pod down**: `{{ $labels.namespace }}/{{ $labels.pod }}` terminated with reason `{{ $labels.reason }}`. See the [runbook](https://runbooks.example.com/pod-down) for remediation steps. ``` To add annotations to a monitor: 1. Define the resource definition for your monitor, and add an `annotations` section: ```yaml theme={null} api_version: v1/config kind: Monitor spec: # Required name of the monitor. Can be modified after the monitor is created. name: Instance is down # PromQL query. If set, you can't set graphite_query. prometheus_query: up{env="prod", instance="kubernetes", exported_job="JOB123"} ==0 # Annotations are visible in notifications generated by this monitor. # You can template annotations with labels from notifications. annotations: summary: 'Instance {{ $labels.instance }} is down' description: 'Container {{ $labels.namespace }}/{{ $labels.pod }}/{{ $labels.container }} terminated with {{ $labels.reason }}.' value: 'Current value: {{ $value }}' runbook: 'http://default-runbook' ``` 2. Apply the changes: ```shell /FILE_NAME/ theme={null} chronoctl apply -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of your notifier YAML file. To add annotations to a monitor: 1. Create a monitor with Terraform by using the `chronosphere_monitor` type followed by a name in a resource declaration: ```terraform theme={null} resource "chronosphere_monitor" "instance_down" { name = "Instance down" # Reference to the collection the alert belongs to collection_id = chronosphere_collection.infra.id # Override the notification policy. # By default, uses the policy from the previously stated collection_id. notification_policy_id = chronosphere_collection.infra_testing.id # Arbitrary set of labels to assign to the alert labels = { "priority" = "sev-1" } # Arbitrary set of annotations to include in alert notifications annotations = { summary = "Instance {{ $labels.instance }} is down" description = "Container {{ $labels.namespace }}/{{ $labels.pod }}/{{ $labels.container }} terminated with {{ $labels.reason }}." value = "Current value: {{ $value }}" runbook = "http://default-runbook" } ... } ``` 2. In the `annotations` section, define the annotations for your monitor. 3. Run `terraform apply` to create the monitor resource: ```shell theme={null} terraform apply ```