Manage service level objectives

Chronosphere Observability Platform provides a structured interface for creating and editing service level objectives (SLOs).

This page provides a reference to this management interface. For conceptual information and best practices in designing SLOs within Observability Platform, see Design service level objectives.

Create a new SLO

When you create a new SLO, Observability Platform creates new metrics that it uses in the SLO’s reporting, alerting, and visualizations. These metrics are prefixed with lens:slo, and you can also query and chart them in your own dashboards.

To create a new SLO:

Edit an SLO

To edit an existing SLO:

Define an SLO

The SLO Definition drawer (also called the Create SLO drawer when creating a new SLO) provides the following sections, each containing options that define the SLO’s parameters. The SLO preview drawer and Code Config tab update as you fill or change the SLO Definition.

SLO information

In the SLO information section, complete these fields:

• Name: The SLO name.
• Owner: The service or collection that owns this SLO.
• Description: User-defined text to describe this SLO’s purpose. This appears in the SLO page’s SLO information section. Use the description to describe to other users what this SLO measures and which downstream users or systems might be affected if the SLO is breached.
• Runbooks: A name and URL for any runbooks used when this SLO triggers. These are displayed as links in the SLO page’s SLO information section.

Alerting

In the Alerting section, complete these fields:

• Alerting is enabled by default. Toggle Alerting enabled to disable alerts on this SLO.

  • Select a Notification policy. Observability Platform then displays the selected policy’s details.

    • When using the Default Policy, this section displays the policy defined for the selected Owner.
    • When using Select Policy, you can choose a different policy than the default.

  • Customize the Burn rate alert configuration, if necessary. This configuration is hidden if alerting is disabled.

    Burn rate alert configuration sets the criteria that determine when the SLO triggers alerts, and of which severity the alerts report. The default burn rate definition applies industry best practices for error budget consumption.

    For example, when your error budget consumption reaches 2% over the last 1h (one hour) Long window and the error rate is still high over the last 5m (five minute) Short window, the SLO fires a critical Severity alert. When the problem no longer exists over the last five minutes, the alert resolves.

    For a full explanation, see Multiwindow, Multi-Burn-Rate Alerts (opens in a new tab).

    You can add an optional Notification label to these alerts, and can add additional criteria by clicking + Add row.

SLO definition

Create the SLO definition, which defines the core criteria the SLO measures. You can provide a Code SLO definition or a Builder definition.

Code definition

In an Code definition, you define what the SLO measures by entering queries.

1. Define whether the Query type returns Errors or Successes.

2. Enter a query that returns the number of errors (Error query) or successes (Success query).

3. Optionally use the following template variables when applicable:

   • {{.Window}}: Use this variable in place of the time interval to dynamically assign the time interval value on the SLO details page. This placeholder computes the optimal window size to fulfill this SLO based on the input time window and burn rates. Using the default values should normally resolve to 5m (five minutes).

     If your query has a rate, you should use {​{​.Window}}. However, gauges can’t use {​{​.Window}}

   • {{.GroupBy}}: Use this variable in place of group by statements in the query when you want a column for each label name defined in the Dimensions section. This placeholder substitutes all of the unique values in dimensions and signal groupings as a comma-separated list. It provides a place that defines the unique values and reduces mismatched queries.

     Observability Platform doesn’t prevent you from managing the two lists without {{.GroupBy}}, but the lists should be identical in the error or success queries and total queries. Those lists should also match the lists in dimensions and signal groupings.

     If your query has a by (...) clause, use by ({​{​.GroupBy}}).

   • {​{​.AdditionalFilters}}: Use this variable in place of long lists of selectors in your SLO queries. This placeholder substitutes all the filters added in the Additional filters section.

     This allows both sharing a single list of filters for both queries if the list is long. {​{​.AdditionalFilters}} can also help when templating SLOs in configuration as code workflows, because you can provide different values based on inputs without needing to directly manipulate the query.

     Observability Platform doesn’t block you from managing the two lists of selectors in your PromQL queries. However, if additional filters are added to the Additional filters section, it’s expected that you’ll use the variable at least once.

     If your queries have a metric{...} where ... is identical, consider using metric{{.AdditionalFilters}}}.

   For example:

   sum by ({{.GroupBy}})(rate(metric[{{.Window}}])

   When cluster and namespace are used as dimensions, the effective query is:

   sum by (cluster, namespace)(rate(metric[5m])

Builder definition

In a Builder definition, you provide only an objective and a service, and Observability Platform then creates the definition for you.

1. Add a percentile-based value for the Objective.
2. Select a Service to measure.
3. Provide additional details about the Type, Endpoints, and Error types.

Dimensions, signals, and filters

Refine your query using Dimensions, signals, and filters.

Use Dimensions to generate a time series per combination of labels entered.

1. Toggle Alert by series to create alerts for each time series in the selected metric.

2. Enter a Label name.

3. Select the Use as signal checkbox to create a signal.

   The signal indicates which labels to alert on. For example, if the base query is sum by (cluster) (rate(metric_name{})), you can add dimensions to make the effective query sum by (cluster, namespace, instance) (rate(metric_name{})) but only have cluster and namespace added as signals to get an alert for each cluster and namespace combination.

4. Add Additional filters to reduce the number of metrics used by the SLO.

   To add a filter:

   1. Click the Add label filter field.
   2. Enter a label, select an operator, and enter a value.
   3. Click the check icon to add the filter, or the close icon to cancel.

   To remove a filter from the Add label filter field, click the close icon on the chip that represents the filter.

Labels and annotations

Add Labels and annotations to provide context for the SLO.

• SLO labels: Add labels to this SLO for use in searches or pinned scopes.
• Annotations: Key/value pairs that add information about the SLO.

SLO preview

Use the SLO preview drawer to ensure the SLO definition meets your specifications. The charts within the preview drawer are the same as those displayed on the SLO’s page after you create or update the SLO. Observability Platform regenerates these preview charts as you modify fields in the SLO Definition drawer.

As you iterate on your SLO’s design, consult these tabs to confirm that the results align with your expectations.

• The SLI tab charts Total requests and Errors over the selected time range.

• The SLO tab charts service availability over the selected time range.

  Toggle Simulate alerts to test your conditions against existing data. The chart displays any alerts that would have fired, and the preview reflects the SLO’s signal grouping, dimensions, and burn rate configuration.

  Use the Show alert durations toggle to display the time range over which the alert would have been active.

  When you make changes to the SLO, click the ** refresh button in the time range selector to run the alerts simulation again.

The preview drawer also indicates the number of new time series that Observability Platform will generate when you save the SLO. It also links to the Telemetry Usage Analyzer for further analysis of the SLO’s expected usage impact.

Delete an SLO

Avoid unintentional budget resets

Changes you make to your SLO’s definition can also unexpectedly change its SLI’s definition, which resets your SLO’s error budget. These unintentional budget resets can destructively change how your SLO represents the service’s performance against its goal.

Changes to these fields cause budget resets:

• Name
• Queries
• Dimensions
• Signals
• Additional filters (label filters)
• SLO labels

When you change the values of these fields, you also change the data emitted from your SLO, which in turn changes how your SLI calculates your error budgets for the SLO’s time window. This results in two sets of budgets, one from before the change and one after the change, being displayed until a full time window has elapsed.

Ensure that a budget reset is an acceptable outcome before you change these fields in an existing SLO.

When changing Queries or Additional filters, you might also change the number of budgets being tracked. This could cause new budgets to appear or old budgets to stop being updated, depending on the change you make.

Configure SLOs with code

Changes you make in the Visual Editor of the Create SLO or SLO Definition drawer are immediately reflected in the Code Config tab, which displays the SLO’s representation in code as either a Terraform resource, a Chronoctl YAML definition, or a JSON object compatible with the Chronosphere API.

You can use the Visual Editor to define changes to an SLO and then Copy or Download its code representation from the Code Config to apply it through these other configuration tools. If an SLO is managed by Terraform, you can modify it only by modifying its Terraform resource.

For details about Observability Platform’s configuration as code features, see Use a GitOps workflow.