Skip to main content
Chronosphere Observability Platform provides a structured interface for creating and editing service level objectives (SLOs). This page provides a reference to this management interface. For conceptual information and best practices in designing SLOs within Observability Platform, see Design service level objectives.

View overall SLO status

You can view a list of SLOs to identify if any are breaching their limits. You can also filter the list to narrow your view to specific keywords, team, or owner. To view a list of existing SLOs:
  1. In the navigation menu, click Go to Admin.
  2. Select System Overview > SLOs.
To filter the SLO list, use one of these methods:
  • Enter text into the Search SLOs search field to filter by name
  • Use the Select an owner dropdown to filter by the SLO’s owning collection or service
  • Use the Select a team dropdown to filter by the SLO’s assigned team.
The SLO table contains the following information:
  • Alert state: The number of active alerts for the SLO, shown as a badge. The badge displays the icon for the most severe active state, with a text label listing all active alert counts. For example, an SLO with two active critical alerts and one warning alert shows the critical icon with the label 2 Critical, 1 Warning.
    IconStateDescription
    CriticalSLO has at least one active critical alert.
    WarningSLO has at least one active warning alert, and no critical alerts.
    MutedSLO’s alerts are muted.
    PassingSLO has no active alerts.
    DisabledAlerting is disabled for this SLO.
  • Name: The SLO’s name.
  • Objective: The objective defined for this SLO.
  • Alerting Enabled: Whether or not alerting is enabled for this SLO.
  • Owner: The service or collection that owns this SLO.
  • Team: The team responsible for the Owner.
  • Source: This SLO’s creation method.
    Users can modify Terraform-managed resources only by using Terraform. Learn more.
The row for each SLO in the list also includes a three vertical dots icon that provides quick access to SLO creation, editing, and deletion. Click the icon to select one of the following options:
  • Duplicate: Click to open the SLO create drawer, populated with the information used to create the existing SLO. Configure the new SLO and then click Save to create the new SLO.
  • Edit: Click to update your SLO using the edit drawer.
  • Delete: Delete the SLO.

View an SLO

Click the name of any SLO in the list to open its page, which is similar to a dashboard and visualizes important metrics related to one or more services.

SLO menu

An SLO page’s menu provides access to features that modify the SLO’s behavior:
  • Events: Click to open the Display events drawer. Select the checkboxes for the events you want to display, and then click Save.
  • Mute: Click to create a muting rule for this SLO. If a muting rule is already active for an SLO, a banner indicates the active muting rule and its expiration.
The menu also includes a link to the documentation and a three vertical dots icon.Click the three vertical dots to select one of the following options:
  • Duplicate: Click to open the SLO create drawer, populated with the information used to create the existing SLO. Configure the new SLO and then click Save to create the new SLO.
  • Edit: Click to update your SLO using the edit drawer.
  • Version history: Review previous versions of this SLO’s configuration.
The menu’s time range selector displays the current time range applied to the SLO’s visualizations and lets you define a new time range. You can also select the SLO’s time range by clicking and dragging across the time span you want to define in any of the SLO’s visualization charts.

Active alerts list

The Active alerts list card shows all currently active alert instances for the SLO. Each entry links to the corresponding alert details page. The card header includes the number of active alerts, for example, Active alerts list (2).If the SLO has no active alerts, the card displays This SLO does not have any active alerts.

SLO details

The SLO details section provides a high-level view of the SLO’s overall health, and indicates whether your SLO is meeting its objective or has breached its target.
  • Availability target: The SLO’s currently defined objective.
  • Reporting status: If the SLO triggers alerts, or if its error budgets are depleted or low, Observability Platform displays additional indicators to summarize these major issues.
The following charts visualize performance against the SLO’s defined limits:
  • Availability: Availability results based on the SLI’s rate definition.
  • Error budget: The SLO’s remaining error budget over its defined time window.
As with all charts on the SLO view, hold the pointer over the chart to reveal three vertical dots. Click the icon to open the chart’s query in Metrics Explorer, add the chart to a dashboard, or investigate it using Metrics DDx. See From another page to add the chart to a notebook, and common panel elements for an explanation of the other available tools.

Reporting status

If an SLO is breached or close to being breached, the SLO page displays a Reporting status that’s otherwise hidden from view. This status contains chips for the SLO’s triggered alerts, depleted error budgets, and error budgets that are close to depletion.If the reporting status is visible for an SLO, you should immediately begin investigating the causes for the statuses it reports.

SLO alerting

If a low-error-rate SLO alert triggers, the alert can continue to trigger for up to the configured long window for hours after the resolution of the issue that caused the alert. How long the alert keeps triggering depends on the rate of decrease in the error budget.

Series

Use the Series subsection’s table to search for or select specific series to view in the Availability and Error budget charts. Each row represents a time series returned in the SLI’s query.The table has the following columns:
  • Alert status: The alert state for that series, shown as a link: Critical, Warning, Muted, Resolved, or Passing. Click the link to go to that series’ alert details page.
  • Columns for labels and values: Each column’s header is the name of a label in that series, and its cells contain that label’s value for that row’s series.
  • Actual: The metric’s value over the SLO’s defined time window.
  • Error budget: The SLO’s remaining error budget for that series. If the cell’s background is red, its value represents a breach of the SLO.
If the SLO uses signal grouping, a Group by signal toggle is available above the table. Enable it to group series rows by their signal group labels.

SLI breakdown

The SLI breakdown section consists of charts that visualize your service level indicators, which are based on the SLO’s definition. For more information, see Define an SLO.These charts include:
  • Total requests: A visualization of the SLI’s total query, representing the total requests to the service.
  • Errors: A visualization of the SLI’s error query.

Burn and error rates

The Burn/Error rates section consists of charts that visualize the error budget burn rate and the rate of reported errors. Burn rate calculations are based on the SLO’s definition. For more details, see Define an SLO.You can adjust the window used for visualizations, which can be 1h, 6h, 1d, or 3d.

Change events

Change events are required for SLO history.
If this service uses change events, those events are graphed in this section. This includes events generated by this SLO and also events added by other features to connected services.

SLO information

The SLO information section provides a user-defined Description of the SLO and relevant Runbook links, as defined in the SLO information section of the create drawer.Related queries depend on features enabled in your tenant. In addition, the SLO must be owned by a service, not a collection. When clicked, the links open in a new tab and populate the page with a query based on the selected SLO. These links include the following:

Ownership

The Ownership section displays the SLO’s Owner, which is a service or collection. Its Notification policy links to the SLO’s selected notification policy.

Labels and annotations

Labels are key-value pairs that filter the SLO to specific telemetry. For example, you might have a service with a label of service and a value of payment-gateway. These values display sequentially.Annotations are key-value pairs that provide additional context for an SLO, such as runbook links or descriptions. Annotation values display on the SLO details page.Annotation values support Markdown formatting, including bold text (**text**), inline code (`code`), named links ([label](url)), and plain HTTP URLs.

Service dashboards

Observability Platform generates a list of Service dashboards based on the dashboards attached to the service that owns the SLO.

Create a new SLO

When you create a new SLO, Observability Platform creates new metrics that it uses in the SLO’s reporting, alerting, and visualizations. These metrics are prefixed with lens:slo, and you can also query and chart them in your own dashboards. From Chronosphere Lens, you can also create SLOs on a service page or with Edit SLO config on Service Configuration without starting from the global SLO list. To create a new SLO:
  1. In the navigation menu, click Go to Admin.
  2. Select System Overview > SLOs.
  3. Click Create SLO. This opens the Create SLO drawer to the Visual Editor by default.
  4. Fill each required field and any optional fields that you want. For a configuration reference, see Define an SLO.
  5. To save the SLO, click Save.
Users can modify Terraform-managed resources only by using Terraform. Learn more.

Edit an SLO

To edit an existing SLO:
To edit an existing SLO from the list of all SLOs:
  1. In the navigation menu, click Go to Admin.
  2. Select System Overview > SLOs.
  3. In the list, hold the pointer over the row of the SLO you want to edit.
  4. Click the three vertical dots that appear in the SLO’s row, and then click Edit.
To edit an SLO from its page, click the three vertical dots in the SLO’s navigation menu, and then click Edit.Both methods open the Visual Editor tab of the SLO Definition drawer, which provides a form interface for configuring the SLO. For configuration-as-code workflows, see Configuring SLOs with code.
Users can modify Terraform-managed resources only by using Terraform. Learn more.

Define an SLO

The SLO Definition drawer (also called the Create SLO drawer when creating a new SLO) provides the following sections, each containing options that define the SLO’s parameters. The SLO preview drawer and Code Config tab update as you fill or change the SLO Definition.

SLO information

Changes you make to your SLO’s Name can also unexpectedly change its SLI’s definition, which causes an error budget reset. An unintentional budget reset can destructively change how your SLO represents the service’s performance against its objective.Ensure that a budget reset is an acceptable outcome before you change this field in an existing SLO.
In the SLO information section, complete these fields:
  • Name: The SLO name. Observability Platform generates a slug from the name when you first create the SLO. The slug is the stable identifier used in API, Chronoctl, and Terraform operations. Changing the name after creation does not change the slug.
  • Owner: The service or collection that owns this SLO.
  • Description: User-defined text to describe this SLO’s purpose. This appears in the SLO page’s SLO information section. Use the description to describe to other users what this SLO measures and which downstream users or systems might be affected if the SLO is breached.
  • Runbooks: A name and URL for any runbooks used when this SLO triggers. These are displayed as links in the SLO page’s SLO information section.

Alerting

In the Alerting section, complete these fields:
  • Alerting is enabled by default. Toggle Alerting enabled to disable alerts on this SLO.
    • Select a Notification policy. Observability Platform then displays the selected policy’s details.
      • When using the Default Policy, this section displays the policy defined for the selected Owner.
      • When using Select Policy, you can choose a different policy than the default.
    • Customize the Burn rate alert configuration, if necessary. This configuration is hidden if alerting is disabled. Burn rate alert configuration sets the criteria that determine when the SLO triggers alerts, and of which severity the alerts report. The default burn rate definition applies industry best practices for error budget consumption. For example, when your error budget consumption reaches 2% over the last 1h (one hour) Long window and the error rate is still high over the last 5m (five minute) Short window, the SLO triggers a critical Severity alert. When the problem no longer exists over the last five minutes, the alert resolves. For a full explanation, see Multiwindow, Multi-Burn-Rate Alerts. You can add optional Notification labels (key-value pairs) to these alerts. Notification labels attach to the alert when a burn rate fires, and notification policies can use them to route different burn rate severities to different notifiers. Add additional burn rate criteria by clicking + Add row.

SLO definition

Changes you make to your SLO’s Queries can also unexpectedly change your SLI’s definition, which causes an error budget reset. An unintentional budget reset can destructively change how your SLO represents the service’s performance against its objective.Ensure that a budget reset is an acceptable outcome before you change these fields in an existing SLO.
Create the SLO definition, which defines the core criteria the SLO measures.
  1. Define the Objective (%) as a percentile value with up to four decimal places. For example, 99.9995.
  2. Define the Time window using standard Chronosphere time unit syntax. The default and recommended value is 4w (4 weeks).
  3. Select the SLO’s Measurement type.
    • Error ratio SLOs measure the objective against the percentage of measurements that report errors over the entire time window. The SLO measures its error budget as the percentage of error responses remaining in the time window before the objective is breached.
    • Time slice SLOs repeatedly measure intervals, or time slices, within the time window and flags them based on a defined threshold. The overall objective then measures the ratio of failed time slices rather than the total number of errors. For instance, a time slice SLO might flag one-minute time slices where availability fails to reach a given threshold, and the overall objective is measured against the percentage of failed time slices over the entire time window. The SLO measures its error budget as the remaining amount of time during which time slice failures would breach the objective.
  4. Define whether the Query type returns Errors or Successes.
  5. Enter a query that returns the number of errors (Error query), successes (Success query), or rate of failures during a time slice (Time slice definition). For error ratio SLOs, also enter a Total query that returns the total number of events. The total query is required for all error ratio SLOs. In configuration-as-code workflows, these fields correspond to bad_query_template (errors), good_query_template (successes), and total_query_template (total).
  6. Conditionally use the following template variables:
    • {{.Window}}: Use this variable in place of the time interval to dynamically assign the time interval value on the SLO details page. This placeholder resolves to 5m (five minutes), which is the recording rule interval used by SLO calculations. When to use it: Gauges can’t use {​{​.Window}}. You should otherwise use {​{​.Window}} in all of your queries to allow your SLO to automatically use the best window sizes.
    • {{.GroupBy}}: Use this variable in place of group by statements in the query to create a column for each label name defined in the Dimensions section. This placeholder substitutes all of the unique values in dimensions and signal groupings with a comma-separated list of the label names in the Dimensions section. It provides a place that defines the unique values and reduces mismatched queries. Observability Platform doesn’t prevent you from managing the two lists without {{.GroupBy}}, but the lists should be identical in the error or success queries and total queries. Those lists should also match the lists in dimensions and signal groupings. When to use it: If your query has a by (...) clause, use by ({​{​.GroupBy}}). For example, if you define dimensions or signals, your query likely contains an aggregate function such as sum by (), and you should pass {{.GroupBy}} as its parameter.
    • {​{​.AdditionalFilters}}: Use this variable in place of long lists of selectors in your SLO queries. This placeholder substitutes all the filters added in the Additional filters section. This allows both sharing a single list of filters for both queries if the list is long. {​{​.AdditionalFilters}} can also help when templating SLOs in configuration as code workflows, because you can provide different values based on inputs without needing to directly manipulate the query. Observability Platform doesn’t block you from managing the two lists of selectors in your PromQL queries. However, if additional filters are added to the Additional filters section, it’s expected that you’ll use the variable at least once. When to use it: Use {{.AdditionalFilters}} when you’ve defined additional filters. If your SLO is defined in Terraform, this can help you template both your filters and queries. For example, if your query has a metric{...} where ... is identical, consider using metric{{.AdditionalFilters}}.
    • {{.TimeSlice}}: Use this variable to reference the SLO’s time slice interval value in the query. This resolves to the configured time slice size (1m or 5m). Available only in time slice SLOs.
    For example:
    sum by ({{.GroupBy}})(rate(metric[{{.Window}}]))
    
    When cluster and namespace are used as dimensions, the effective query is:
    sum by (cluster, namespace)(rate(metric[5m]))
    
    In a time slice SLO, use {{.TimeSlice}} to reference the slice duration. Both {{.Window}} and {{.TimeSlice}} are valid in time slice queries and serve different purposes. {{.Window}} is the recording interval and {{.TimeSlice}} is the slice size:
    sum by ({{.GroupBy}})(rate(metric[{{.TimeSlice}}]))
    
  7. In time slice SLOs, complete the fields within the sentence that defines the objective:
    • Choose an interval from the first dropdown, which defaults to 1 minute. Available intervals are 1 minute and 5 minutes.
    • Choose an operator from the second dropdown, which defaults to greater than or equal to (>=).
    • Enter a Threshold that, when combined with the operator, determines whether the SLO considers a time slice to be a success.

Dimensions, signals, and filters

When you add or remove Dimensions and Additional filters from your SLO, Observability Platform can also unexpectedly change its SLI’s definition, which causes an error budget reset. An unintentional budget reset can destructively change how your SLO represents the service’s performance against its objective.Ensure that a budget reset is an acceptable outcome before you change these fields in an existing SLO.
Refine your query using Dimensions, signals, and filters. Use Dimensions to generate a time series per combination of labels entered.
  1. Toggle Alert by series to create alerts for each time series in the selected metric. When enabled, each individual series produces its own signal for alerting purposes. In configuration-as-code workflows, this corresponds to the signal_grouping.signal_per_series field. Alternatively, add labels with the Use as signal checkbox selected to group series into signals by specific label combinations. This corresponds to the signal_grouping.label_names field. These two options are mutually exclusive.
  2. Enter a Label name.
  3. Select the Use as signal checkbox to create a signal. The signal indicates which labels to alert on. For example, if the base query is sum by (cluster) (rate(metric_name{})), you can add dimensions to make the effective query sum by (cluster, namespace, instance) (rate(metric_name{})) but only have cluster and namespace added as signals to get an alert for each cluster and namespace combination.
  4. Add Additional filters to reduce the number of metrics used by the SLO. To add a filter:
    1. Click the Add label filter field.
    2. Enter a label, select an operator, and enter a value.
    3. Click the check icon to add the filter, or the close icon to cancel.
    To remove a filter from the Add label filter field, click the close icon on the chip that represents the filter.

Labels and annotations

Changes you make to your SLO’s Labels can also unexpectedly change its SLI’s definition, which causes an error budget reset. An unintentional budget reset can destructively change how your SLO represents the service’s performance against its objective.Ensure that a budget reset is an acceptable outcome before you change this field in an existing SLO.
Add Labels and annotations to provide context for the SLO.
  • SLO labels: Add labels to this SLO for use in searches or pinned scopes.
  • Annotations: Key/value pairs that provide additional context for the SLO, such as runbook links or descriptions. Annotation values display on the SLO details page and in notifications generated by this SLO. Annotation values support templating with labels from the notification signal. Annotation values support Markdown formatting, including bold text (**text**), inline code (`code`), named links ([label](url)), and plain HTTP URLs.

SLO preview

Use the SLO preview drawer to ensure the SLO definition meets your specifications. The charts within the preview drawer are the same as those displayed on the SLO’s page after you create or update the SLO. Observability Platform regenerates these preview charts as you modify fields in the SLO Definition drawer. As you iterate on your SLO’s design, consult these tabs to confirm that the results align with your expectations.
  • The SLI tab charts Total requests and Errors over the selected time range.
  • The SLO tab charts service availability over the selected time range. Toggle Simulate alerts to test your conditions against existing data. The chart displays any alerts that would have triggered, and the preview reflects the SLO’s signal grouping, dimensions, and burn rate configuration. Use the Show alert durations toggle to display the time range over which the alert would have been active. When you make changes to the SLO, click the ** refresh button next to the time range selector to run the alerts simulation again.
The preview drawer also indicates the number of new time series that Observability Platform will generate when you save the SLO. It also links to the for further analysis of the SLO’s expected usage impact.

Delete an SLO

To delete an existing SLO from the list of all SLOs:
  1. In the navigation menu, click Go to Admin.
  2. Select System Overview > SLOs.
  3. In the list, hold the pointer over the row of the SLO you want to edit.
  4. Click the three vertical dots that appear in the SLO’s row, and then click Delete.
To delete an SLO from its page:
  1. Click the three vertical dots in the SLO’s navigation menu, and then click Edit.
  2. Scroll to the end of the SLO Definition drawer and click Delete SLO.
  3. In the confirmation dialog, click Delete SLO to confirm.
Users can modify Terraform-managed resources only by using Terraform. Learn more.

Avoid unintentional budget resets

Changes you make to your SLO’s definition can also unexpectedly change its SLI’s definition, which resets your SLO’s error budget. These unintentional budget resets can destructively change how your SLO represents the service’s performance against its goal. Changes to these fields cause budget resets:
  • Name
  • Queries
  • Dimensions
  • Signals
  • Additional filters (label filters)
  • SLO labels
When you change the values of these fields, you also change the data emitted from your SLO, which in turn changes how your SLI calculates your error budgets for the SLO’s time window. This results in two sets of budgets, one from before the change and one after the change, being displayed until a full time window has elapsed. Ensure that a budget reset is an acceptable outcome before you change these fields in an existing SLO. When changing Queries or Additional filters, you might also change the number of budgets being tracked. This could cause new budgets to appear or old budgets to stop being updated, depending on the change you make.

Configure SLOs with code

Changes you make in the Visual Editor of the Create SLO or SLO Definition drawer are immediately reflected in the Code Config tab, which displays the SLO’s representation in code as either a Terraform resource, a Chronoctl YAML definition, or a JSON object compatible with the Chronosphere API. You can use the Visual Editor to define changes to an SLO and then Copy or Download its code representation from the Code Config to apply it through these other configuration tools. If an SLO is managed by Terraform, you can modify it only by modifying its Terraform resource. For details about Observability Platform’s configuration as code features, see Use a GitOps workflow.