Chronosphere Lens service configuration

Chronosphere Lens provides a view into your applications based on services (opens in a new tab). If a service doesn't match what you expect, you can edit the service configuration in Observability Platform to include or remove services from your display. You can also edit the visual presentation of your data.

Understand service configuration

The Service configuration page lets you configure how to discover what services exist in your environment based on your telemetry data, and what visualizations to present on your services pages based on their data.

All users can view service configurations and preview changes. Only users with SysAdmin permissions can save new or edit existing service configurations.

View service configuration

In the navigation menu, click Go to Admin and then select Platform > Service Configuration.

If there are unsaved changes, the Review & apply changes button indicates how many unsaved changes there are. Changed but not saved jobs are indicated by an alert icon.

Discovery jobs

Discovery jobs are templates that match metrics to a discovered service. Chronosphere provides some common discovery jobs. Create your own jobs or copy and customize existing jobs for your environment

Use the Search jobs box to find a specific template.

• Status: Toggle the switch to enable or disable the job.
• Managed by: Jobs are manged by Chronosphere (non-editabl), or by the user (editable).
• Job name: The user-readable discovery job name.
• Type: Users can create jobs to discover services based on metrics (Metrics) or change events (Events) types. Traces are managed by Chronosphere and not user creatable or editable. Traces is used by the Trace Services Based on Service Attribute job, and is available only to customers with a Tracing license. Users can enable or disable the discovery job.
• Service label mapping: Labels identifying the service name. Provided by the the service label or instance label + identifying label (Service template label) from the template form.
• Services discovered: The estimated maximum number of discovered services. This number can be significantly different (often higher) than the actual number of discovered services, depending on merge rules.

Global mapping

Global mappings let services be discovered by jobs using metrics that have a pod/instance label but no obvious service labels.

Discovery jobs can be configured to join their pod (opens in a new tab) label with the pod label from one of the global mapping configurations that has a label that's more likely to map to a service name. For example, the kube_pod_info metric from kube-state-metrics (KSM) (opens in a new tab) have a created_by_name which is often a service name. Chronosphere uses those labels to match the physical pods to a discovered service.

User can't edit configurations in the Global mapping tab. These configurations are commonly from standard KSM and OpenTelemetry libraries. Introducing new potentially conflicting configurations here could cause services to be silently ignored by discovery jobs.

• Slug: The slug name of the discovery rule defining the service pod mapping.
• Name: The rule name.
• Service labels: One or more labels that could be a candidate for the service name.
• Pod label: The pod label is usually the name of the pod or instance and is used as a key to map to the service name. These are created during metrics discovery jobs.

Click the three vertical dots icon and then Open in Metrics Explorer to open the query for that service instance in Metrics Explorer.

Create a discovery job

To create a discovery job:

1. On the Service Configuration page, click Create.

2. From the menu, choose Metrics discovery or Change events discovery.

3. Add a Job name.

4. Add Job details:

   MetricsEvents

   For a Metrics discovery job, add:

   • Metric name: Observability Platform queries this metric to find labels which contain candidate service names based on the discovery job configuration.

   • Metric name prefix: A prefix of metric names to used filter and scope metrics this job applies to. This field uses glob syntax to match metric names. When a service is discovered by this discovery job, metrics with this prefix will receive a derived label called lens_service which is set to the discovered service name.

     ⚠️

     This prefix along, with optional target label names in this job must not conflict with the prefix in another enabled discovery job.

   • Optionally, add a Target name and Target glob. These fields are a label name and value glob expression to further scope what metrics to add the lens_service label to. It's used to help scope the discovery job and avoid conflicts when you need to configure different discoveries on metrics with the same prefix.

   • Service mapping:

     Focus your template using one of the following groups:

     • Service label: A label containing the service name. This label must be found on the Metric name previously specified. This is the preferred option for when you already have a good service name label on your metrics.
     • Pod mapping: Pod mappings are an advanced option. Use if you don't have an obvious label to use as the service name, but you have a pod or instance label that could join with a global mapping that has a service name.

       • Instance label: The label containing the instance or pod name. Label values are used as keys to map to the service name. For example, GRPC metrics have an instance label of instance and an identifying label of grpc_service.

       • Transform regex: A regular expression that modifies the value in the instance label to match a value in the service instance mappings. This is often used to remove a pod name's prefix. Can be (+.) if no trimming is required.

       • Transform replacement: The regular expression replacing the original label, allowing modification of the value in the instance label so that it matches a value in the service instance mappings. You can use symbols like $1 to insert regular expression capture groups.

       • Identifying label: A label containing information that uniquely identifies which service the time series belongs to. The identifying label can't be the instance label or other ephemeral label. It must be static, finite, and uniquely identify the service. For example, an identifying label value of X that indicates it belongs to service1 must always map to service1. The value X can't map to another service.

         ⚠️

         This mode of service discovery fails if this label is high cardinality.

5. Click Create.

6. Click Review & apply changes.

Edit or delete a discovery job

You can edit or disable existing user-created jobs to ensure your configurations match your services when those services change.

1. Click the three vertical dots and then select Edit config to change or Delete to remove the job.

2. Available edits to job details include:

   • Clear the Enabled checkbox to disable the job.
   • Update the Service label.
   • For metrics discovery, update the Pod mapping fields.

3. Click Stage changes to preview your updated configuration.

4. Review your changes.

5. Click Review & apply changes to save.

When editing a metrics discovery job, you can review the Service discovery query, which is the query used to generate the discovery job. Click the icon to open the query in Metrics Explorer.

Merge rules

When a service doesn't match expectations, create merge rules to correct mismatches. Merge rules define how to group or match service data when service names or labels might not be the same across multiple instances.

Use the Search rules box to find a specific rule, or filter by Rule type.

The Merge rules table displays information that was set during rule creation.

Merge rules apply from the top to the bottom of the list, in Order. A service matches the first rule that applies to it, and won't match additional rules. Use the drag icon to move the rule up or down in the list to change the order that rules apply.

Toggle the Status icon to disable a job.

Preview and apply changes

This table is a list of changes that will take effect when you apply any staged merge rule updates. To display only the staged changes, click the Show diff only toggle. If Review & apply updates displays a 0 in the button, no user-created changes are staged to apply. However, the preview pane can still display staged changes. These changes are due to discovery jobs run by Observability Platform that haven't yet been applied.

Run discovery

To run discovery jobs:

1. Click Review & apply changes to open a larger Review & apply changes drawer. This drawer lets you review a more detailed preview, including any Presentations attached to services. Click the Use freshest data toggle to run the preview without a cache from previous discovery runs.

2. Click Run service discovery. You must have SysAdmin permissions to run service discovery.

In the Review & apply changes drawer, the number of unchanged, added, inactivated, and modified services displays under the drawer title. Select Skip service discovery to wait until the next schedules discovery run.

To discard changes, click Reset changes.

Create a merge rule

To create a merge rule:

1. In the Merge rules tab, click Create, and then select a rule type:
   • Ignore rules remove them from the interface display, and stop collecting data. Discovery never adds additional services matching this rule.
   • Merge rules group services together.
   • Suppress: Use this to prevent noisy service discovery jobs from creating too many ephemeral services in your environment. Adding a Suppress rule for a given Suppressed Discovery Slug means that this discovery job won't discover new services. If a service is discovered by another discovery job, presentations configured to display for the suppressed discovery job will display.
2. Add the following fields:
   • Rule Description: Describe what this service rule does. Displays on the configuration page as Description.
3. The following fields display based on your rule type:
   • General Settings: The Rule description.
   • Rule details: The following fields display in the Details section of the Merge rules page.
     • Source labels: Specify which label's values the rules apply to. The Discovered service name is a source label, and is most commonly selected.
   • Regex: Add a new regular expression to replace the original. On the service configuration page, the original Regex displays, along with the Replacement, which is the value added here.
   • Suppressed Discovery Slug: When using a Suppress rule, add a discovery slug to suppress new services from being generated by that discovery.
     • Exempt Service Names: When using a Suppress rule, don't suppress the listed services, even if they're generated by suppressed discoveries.
4. Click Stage changes. The Service configuration page displays, with your updates displayed.
5. The Preview updates. Preview your changes using the table. The Service name and the Discovery jobs included in that service show what metrics are merged into a given service name. Services and metrics being removed from display have a [-] before them.
6. Click Review & apply changes.

Edit or delete a rule

To edit or delete a service rule:

1. Click the three vertical dots icon.
2. Click Edit rule to change the rule, or Delete to remove the rule.
3. Edit the fields, per their descriptions in the create rules section.
4. Click Stage changes. The Service discovery preview displays your updates.
5. Click Apply changes to save or Reset changes to cancel.

Presentations

Presentations are templates for how service data displays. Observability Platform provides default presentations for your use. You can also create your own custom presentations.

View presentations

To review existing presentations, on the Service Configuration page, click Presentations.

All of your presentations display as cards. The cards are separated into two groups:

• Chronosphere Managed: Default presentations, which can't be edited.
• Custom Presentations: Editable presentations created by users.

Each card displays the following information:

• Name: The presentation name.
• Enabled status: Either Enabled or Disabled.
• Number of Services: The number of services this presentation represents.
• Number of panels: The number of dashboard panels this presentation displays.

Create a presentation

You can create custom presentations for your services. Presentations can use any panel type that a Chronosphere dashboard can use. To create a presentation:

1. In the Presentations page, click Create.
2. Add a Name and Description to identify the presentation.
3. Create conditions the presentation must meet. In the IF service has menu, select a discovery job. The presentation will show on services that have been discovered by this discovery job.
4. Click + Add condition to show more services, or to hide specific services from this presentation.
5. Review the list of services that match the selected conditions. Make changes to the conditions if needed.
6. The Review Content section of the creation page defines how the content displays on the service page.
7. Click Start with existing dashboard if you already have a dashboard created to use. Click Start a new view to create a new dashboard, which opens the dashboard editor in a new tab.

Use an existing dashboard

If you have a dashboard for an existing service, and you're creating a presentation in Lens, you can select the service as a variable. Observability Platform replaces that variable in the JSON with lens_service to make the presentation flexible for all services, which lets you reuse this presentation for multiple services.

To import an existing dashboard:

1. Click Start with existing dashboard.
2. Select a dashboard from the list.
3. Click Save when finished.
4. Click Review & apply changes.

Start a new view

If you select Start a new view, the dashboard editor opens in a new tab.

1. Add panels and variables to your new dashboard.
2. Click Save when finished.
3. Click Save in the Create screen.
4. Click Review & apply changes.

Edit a presentation

To change an existing presentation:

1. On the Service Configuration page, click the three vertical dots and then select Edit presentation, or on the Presentations page, click the card for the service.
2. Make changes to your presentation.
3. Click Save.

Delete a presentation

To delete a presentation, edit the presentation, and then click Delete presentation.