# Administer your tools and users Source: https://docs.chronosphere.io/administer Chronosphere Observability Platform offers tools and guidance for managing users, teams, collections, and your infrastructure-as-code settings. Administrative users focus on observability tools so that developers can use those tools to monitor and maintain the health of their distributed system. As an administrative user, you might focus on one or more of the following tasks. ## Cost control Track telemetry usage against your contract with the [License Overview](/administer/limits-licensing/licensing), which surfaces consumption snapshots and trends for each telemetry type. To investigate what's driving growth, [analyze consumption](/control/consumption/analyze) across partitions and budgets. Then use the [Control Plane](/control/shaping) to shape, sample, and reduce the data you persist. ## Data collection [Ingest telemetry data](/ingest) into Chronosphere Observability Platform, which can include metrics, traces, and change events. Your Collector selection depends on the data you plan to ingest. Chronosphere recommends using the Chronosphere Collector, but can also ingest data from the OpenTelemetry Collector. ## Data usability [Explore your data](/investigate) to understand where you can normalize, sanitize, and optimize telemetry data to make it usable for developers and to filter out less actionable data. [Analyze usage metrics](/overview/types/metrics) and implement aggregation rules to improve the quality and reduce the cardinality of stored metrics data. ## Service level objectives Define [service level objectives](/observe/slo) (SLOs) that measure your services' availability from your users' perspective. Effective SLOs combine user-focused indicators, realistic objectives based on historical performance, and tuned burn rate alerts that help responders react before error budgets are exhausted. [Design service level objectives](/administer/design-slos) to choose the right indicators, set realistic objectives, and configure burn rate alerts for your services. ## Platform usability Create and manage [collections](/administer/collections) to group Observability Platform resources associated with a service or feature you want to monitor. You create [teams](/administer/accounts-teams/teams) to organize accounts into groups and define permissions for sensitive management and administrative operations. A team's accounts have shared responsibility for individual services, which Observability Platform represents as collections. # Accounts and teams Source: https://docs.chronosphere.io/administer/accounts-teams Chronosphere Observability Platform lets you manage user accounts, roles, and team permissions for streamlined access and collaboration. Chronosphere Observability Platform identifies users and services by their *accounts*. As an administrator, you can organize accounts into *teams* and define those teams' permissions. * A [*user account*](/administer/accounts-teams/user-accounts) represents a user's identity in Observability Platform. * A [*service account*](/administer/accounts-teams/service-accounts) represents a service's identity in Observability Platform. * A [*team*](/administer/accounts-teams/teams) organizes accounts into groups and defines permissions for sensitive management and administrative operations. Add accounts to teams based on the permissions you want grant to those accounts. For example, to make a user or service account an *administrator*, assign it to a team that has the SysAdmin role. This grants the account access to special Observability Platform features, including team and account management and assignment. ## View accounts and teams In the navigation menu, click ** Go to Admin** and then select **Platform > Users**. The left-hand column lists all accounts. The list includes icons for additional context about each entry type: | Icon | Description | | :--------------------------: | --------------------------------------------------- | | | An active user account. | | | A service account. | | | An invited user who's yet to confirm their account. | | | A deactivated user account. | A **System Administrator** icon to the right of an entry indicates either a team that has the System Administrator (SysAdmin) role, or a user or service account that's a member of a team with that role. To view details about an account, select its name from the list. To filter by account name, enter part or all of the account's name in the **Search** field. To filter by account type or status, click the filter icon. When you select an account, Observability Platform displays the account's team **Memberships** and **Recent User Actions**. If your user account belongs to a team with the SysAdmin role, this view also provides options to add or remove the account from teams. In the navigation menu, click ** Go to Admin** and then select **Platform > Teams**. The table lists all teams by name and other details about their configuration: * **Role**: The role granted to all accounts in the team. The SysAdmin role grants administrative privileges. * **Source**: The source of this team's configuration. **UI** teams are configured in Observability Platform and can be edited in the app by users with the SysAdmin role. **Terraform** teams are configured exclusively through Terraform. * **Description**: A description of the team. To view a team's [home page](/administer/collections/home), click its name in the list. If your user account is part of a team with the SysAdmin role, you can edit the team's name, role, description, links, and membership from its home page. You can use [Chronoctl](/tooling/chronoctl) to list teams and their member accounts. This output includes the email address associated with each account. If an account's email address is an alphanumeric Observability Platform slug at the `serviceaccount.chronosphere.io` domain, that member is a [service account](/administer/accounts-teams/service-accounts). Other email addresses represent user accounts. To view accounts and teams with Chronoctl, you must authenticate as a user that belongs to a team with the SysAdmin role, or as an unrestricted [service account](/administer/accounts-teams/service-accounts). To list each team and its member accounts in separate YAML documents with [Chronoctl](/tooling/chronoctl), use the `teams list` command. You can also use this output as a template for updating each team separately using Chronoctl. ```shell theme={null} chronoctl teams list ``` The output contains YAML documents separated by three dashes (`---`). Each document contains one team and any of its member accounts. For example, this output lists a team named *Control team* with one user account and one service account as members, and a second team named *DBAs* with two user account members: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Team spec: slug: control-team name: Control team user_emails: - adam@example.com - 4a5842df-4179-47c7-bcc0-d1e6589caaf0@serviceaccount.chronosphere.io --- api_version: v1/config kind: Team spec: slug: dbas name: DBAs user_emails: - adam@example.com - betty@example.com ``` To list all teams with the Chronosphere API, use the [`ListTeams`](/tooling/api-info/definition/operations/ListTeams) endpoint. To view a single team, use the [`ReadTeams`](/tooling/api-info/definition/operations/ReadTeam) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Grant System Administrator (SysAdmin) privileges to a team A user account that's a member of a [team](/administer/accounts-teams/teams) with the `SysAdmin` role has administrative access to Observability Platform features, including the abilities to create teams and user accounts, assign users to teams, and define team permissions. For instructions about assigning roles to a team, see [Add a role to a team](/administer/accounts-teams/teams#add-a-role-to-a-team). # Okta user synchronization Source: https://docs.chronosphere.io/administer/accounts-teams/okta Configure user synchronization between Okta and Chronosphere Observability Platform. [System for Cross-domain Identity Management (SCIM)](https://en.wikipedia.org/wiki/System_for_Cross-domain_Identity_Management) is a standard used to automate exchanges of user identity details between identity systems or Identity Providers (IdP), like Okta. It's used to deprovision, update, and provision users. SCIM integration helps you control user access in Chronosphere Observability Platform. This document applies to existing customers using Okta who want to use SCIM. New customers can configure SCIM when moving to Observability Platform. After migrating to SCIM, Chronosphere Support sends you a list of users not managed by SCIM. On a case-by-case basis, determine if a user should be deactivated, or if the user is still active but has a new email address. Your Chronosphere team manually remediates these issues to complete the migration process. ## Setup Setting up SCIM with Okta requires configuration in both Observability Platform and the Okta dashboard. ### Prepare Observability Platform for SCIM integration Setting up SCIM with Okta requires a user with Okta super administrator access and a user with Observability Platform SysAdmin permissions. These accounts can be the same user. Before proceeding with Okta integration, complete the following steps: 1. Contact Chronosphere Support to enable SCIM integration for Okta Workforce for your application. After Chronosphere Support enables SCIM integration, new users won't be able to access Observability Platform until the setup process is complete. 2. [Create an unrestricted service account in Observability Platform](/administer/accounts-teams/service-accounts#create-an-unrestricted-service-account). You must be a member of a team with the SysAdmin role to create a new service account. For the **New Service Account Name**, Chronosphere recommends a meaningful service account name like `Okta SCIM integration`. 3. Copy the token to a safe place, as it's provided only once, and can't be displayed or recovered later. 4. In Observability Platform, [create a new, distinct team](/administer/accounts-teams/teams) for the purpose of SCIM administration. 5. Assign the [**User Administrator** role to the team](/administer/accounts-teams/teams#add-a-role-to-a-team). For security, Chronosphere recommends only this team be assigned the **User Admin** role, and the role be specifically scoped to only have permission to communicate with the SCIM API, or have access to the service token. 6. Add the service account user you created. ### Configure SCIM integration on Okta Your organization's Okta administrator must configure Okta provisioning integration. Find general instructions for setting up Okta integration in the [Okta documentation](https://developer.okta.com/docs/guides/scim-provisioning-integration-connect/main/). 1. Using an administrator account, sign in to the Okta app to be used for single sign-on (SSO) integration with Observability Platform. 2. Next to your username, click **Admin**. 3. In the left sidebar menu, go to **Applications > Applications**. 4. In the **General** tab, next to **Provisioning**, select **SCIM** and then click **Save**. 5. Click the **Provisioning** tab. 6. Click **Integration**, and then click **Edit**. 7. Enter information for the following fields: * **SCIM connector base URL:** `https://TENANT.chronosphere.io/api/scim/v2`. * **Unique identifier field for users:** Enter `email`. * **Supported provisioning actions:** Select only these values: * **Push New Users** * **Push Profile Updates** 8. For **Authentication Mode**, select `HTTP Header`. 9. For the **Authorization** section's **Bearer** field, copy and paste the service token obtained when creating a service account in the Observability Platform console. 10. Click **Test Connector Configuration** to ensure the integration configuration is correct. If you encounter an error message, review the configuration and try again. 11. Click **Save** to save the configuration. The **Provisioning to App** page displays. 12. Select the **Enable** checkboxes for **Create Users** and **Update User Attributes**. 13. Click **Save**. You must complete both step 7 (select **Push new users**), and step 12 (Enable the checkbox for **Create Users**). Provisioning fails if either of these steps isn't completed. ### Update the Observability Platform default group in Okta After completing SCIM integration setup process and connecting to Observability Platform, sync the existing users in your Okta tenant with Observability Platform. To do this, you must first remove all assigned users and groups, and then reassign them. 1. Sign in to Okta. 2. Click **Admin**. 3. In the left sidebar menu, go to **Applications > Applications**. 4. In the **Provisioning** tab, clear the **Deactivate Users** checkbox. Clear this checkbox before updating the group. 5. Navigate to the **Assignments** tab and then select **Groups**. 6. To remove all **Existing Groups**, click the **X** icon next to each group. 7. Click the **Assign** button and reassign all **Existing Groups**. 8. Navigate to the **Provisioning** tab, and then click **Edit**. 9. Select the **Deactivate Users** checkbox. The SCIM integration setup is now complete and users will be provisioned and deprovisioned by Observability Platform. ### Verify successful provisioning After completing the setup process, verify the provisioning process succeeded. 1. In Okta, navigate to **Dashboard -> Tasks**. 2. Look for failed requests. These display as `Application assignments encountered errors`. 3. Retry any failed requests. If failures persist, contact [Chronosphere Support](/support). It's a good practice to review for failed requests whenever deactivating users or changing users assigned to Observability Platform. Use [Okta Workflows](https://help.okta.com/wf/en-us/Content/Topics/Workflows/use-case-lifecycle-event-notification.htm) to send notifications when provisioning or deprovisioning fails. ## Deprovisioned users When a user is deprovisioned, personal access tokens and any outstanding credentials they had from Observability Platform no longer work. SCIM deactivates users from the time of enablement, but won't deactivate existing users who are no longer assigned to or were removed from the Chronosphere app. Group memberships remain intact, and [service accounts](/administer/accounts-teams/service-accounts) continue to function. Observability Platform doesn't revoke service accounts created by a given user during user deactivation. Service accounts can be used in production, and deactivating them can cause outages. This is consistent with other web services, such as Amazon Web Services and Google Cloud Platform, where service accounts aren't tied to a user. # Personal access tokens Source: https://docs.chronosphere.io/administer/accounts-teams/personal-access-tokens How Chronosphere Observability Platform supplies users more secure access to its APIs, Chronoctl, and Terraform. A user can create a *personal access token* to authenticate their user account with Chronosphere APIs, Chronoctl, and Terraform. When you authenticate with a personal access token, Chronosphere Observability Platform associates the actions you take with your identity and respects the permissions granted to your user account's team. Unlike creating service accounts, you don't need to be a member of a team with the SysAdmin role to create a personal access token. For details about user accounts and teams, see [Accounts and Teams](/administer/accounts-teams). For details about signing in to Observability Platform in a web browser, see [Authenticating with Chronosphere](/overview/authenticating). A personal access token expires and stops working after a defined period of up to 30 days. You can't use a personal access token as an API token for the Collector. To create a persistent identity for a service, define permissions for it, and provide it with an API token, use a [service account](/administer/accounts-teams/service-accounts). ## Create a personal access token To create a personal access token: 1. Click your profile icon from the menu bar and select **My Account**. 2. Click **+ Add Token** to open the **Add Token** window. 3. Enter a name for the personal access token. 4. In the **Token Expiration** dropdown, select a token duration in days. 5. Click **Generate Token** to save the personal access token. 6. Copy the resulting API token. Store the API token securely, because it's as sensitive as your password. If you don't store the API token immediately after creation, you can't view it after navigating away from the **My Account** page. If you lose the API token, you must create a new personal access token. ## Use a personal access token You can use the API token generated by your personal access token with Chronoctl or the Chronosphere API to access the same things you can access in the Observability Platform web app. For details, see the [Chronoctl](/tooling/chronoctl), [Prometheus API](/tooling/prometheus-api), and [Chronosphere API](/tooling/api-info) documentation. ## Revoke personal access tokens Observability Platform invalidates each personal access token and its associated API token upon expiration. You can also manually revoke personal access tokens that you've created. When you revoke a personal access token, Observability Platform removes it and its associated API token immediately without any additional confirmation. To revoke a token: 1. Click your profile icon from the menu bar and select **My Account**. 2. On the row of the token you want to revoke, click ** Revoke**. To revoke all of your tokens: 1. Click your profile icon from the menu bar and select **My Account**. 2. Click **Revoke All**. # Service accounts Source: https://docs.chronosphere.io/administer/accounts-teams/service-accounts An explanation of Chronosphere service accounts and usage. A *service account* provides an identity that a service can use to access Chronosphere APIs, Chronoctl, Terraform, and the Chronosphere Collector. Use service accounts for automated, programmatic interactions, such as CI/CD, ingesting telemetry, or automated queries. Chronosphere Observability Platform attributes all actions that a service takes to its service account. A service account can't access the user interface, only the APIs. Accounts can belong to [*teams*](/administer/accounts-teams/teams). When you create a service account, Chronosphere also creates a non-expiring API token that the service account uses for authentication. Service accounts are distinct from [*user accounts*](/administer/accounts-teams/user-accounts), which provide identities for users who can also authenticate interactively and use Chronosphere Observability Platform. To use tools that require non-interactive authentication, such as Chronoctl, Terraform, or the Chronosphere API, create a temporary [personal access token](/administer/accounts-teams/personal-access-tokens) with the same permissions as your user account. You can also assign service accounts to teams with the `SysAdmin` role, which also grants those service accounts administrative permissions. For details about accounts and teams, see [Accounts and Teams](/administer/accounts-teams). For details about interactive user account authentication, see [Authenticating with Observability Platform](/overview/authenticating). ## View service accounts In the navigation menu, click ** Go to Admin** and then select **Platform > Service Accounts**. The service account list contains the following information: * **Name:** The service account name. * **Service account email:** The unique email user for this service account. * **Created by:** The service account creator. * **Restriction:** The type of [restrictions](/administer/accounts-teams/service-accounts#create-a-restricted-service-account) on this account. Unrestricted accounts don't have a value here. * **Last used:** When this service token was last accessed. This field can be inaccurate if more than 1,000 service accounts are active. * **Metric label:** For restricted accounts, the key-value label pair restriction. To return all service accounts and their label restrictions, use the [Chronoctl](/tooling/chronoctl) `service-accounts list` command. For example, to return all service accounts in JSON format, run: ```shell theme={null} chronoctl service-accounts list --output=json ``` The command returns the following fields for each service account: * `id`: The unique identifier for the service account. The `slug` field contains the same value. * `name`: The service account name. * `permission`: For restricted accounts, the access level: `READ_PERMISSION`, `WRITE_PERMISSION`, or `READWRITE_PERMISSION`. Omitted for unrestricted accounts. * `labels`: For restricted accounts, the key-value label pair restrictions. The command doesn't return the account email or creator. The default `table` output shows only the `id`, `slug`, and `name` fields. To complete this action with the Chronosphere API, use the [`ListServiceAccounts`](/tooling/api-info/definition/operations/ListServiceAccounts) endpoint. The endpoint returns the following fields for each service account: * `slug`: The unique identifier for the service account. * `name`: The service account name. * `email`: The unique email user for this service account. Read-only. * `created_at`: When the service account was created. * `updated_at`: When the service account was last modified. * `unrestricted`: Whether the account has full access to all Chronosphere APIs. * `metrics_restriction`: For restricted accounts, the metric access restriction. Contains a `permission` level of `READ`, `WRITE`, or `READ_WRITE`, and optional `labels` that limit access to metrics with matching label names and values. The `token` field is returned only in the `CreateServiceAccount` response, so it's empty in list results. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. If you have a service account token, and don't know which account it belongs to, use the following Chronoctl command: ```shell theme={null} chronoctl auth whoami --api-token=TOKEN ``` Replace *`TOKEN`* with your service account token. ## Create a service account New service accounts can be one of the following types, each configured differently: * An *unrestricted service account*, which grants full access to all Chronosphere APIs and entities, including administration and monitoring features. * A *restricted service account*, which grants certain permissions to access only telemetry data, and optionally limiting displayed metrics to those with one or a pair of specific, case-sensitive label names and values. To use a service account with Chronoctl or Terraform, you must create an unrestricted service account. To use a service account with the Chronosphere Collector, create a restricted write-only service account. ### Create an unrestricted service account A service account can access the entire Chronosphere API, but this access doesn't necessarily grant it all possible privileges. You can also assign the `SysAdmin` role to the team that the service account belongs to, but doing so is required only if the service account needs to create, modify, and delete accounts, teams, and [certain resources](/administer/accounts-teams/teams#assign-team-roles-and-permissions). To make an unrestricted service account read-only, remove it from the **Default** team and any other team that grants the `Editor` or `SysAdmin` role. The account retains read-only access to all Chronosphere APIs. For more information, see [Assign team roles and permissions](/administer/accounts-teams/teams#assign-team-roles-and-permissions). You **must** be a member of a team with the `SysAdmin` role to create a new service account. Select from the following methods to create an unrestricted service account. To create an unrestricted service account: 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Service Accounts**. 2. Click **+ Service Account**. This opens the **Add Service Account** dialog. 3. In the **Service Account Name** field, enter a descriptive name for the service account. 4. In the **Service Account Type** section, select **Unrestricted**. 5. Click **Save**. After creating the service account, Chronosphere displays its secret token. This is the only time Observability Platform displays the service account's token. Store it securely. If you lose the token, you must delete and recreate the service account to generate a new token. To create and manage service accounts with [Chronoctl](/tooling/chronoctl), the account you use must either be a member of a team with the [`SysAdmin` role](/administer/accounts-teams/user-accounts#grant-system-administrator-sysadmin-privileges-to-users) or an unrestricted service account. 1. Generate a templated service account resource with the `chronoctl service-accounts scaffold` command: ```shell theme={null} chronoctl service-accounts scaffold > service-account.yml ``` Replace `service-account.yml` with any valid filename. 2. Edit the generated template to configure the new service account. Remove any fields that you don't need to configure. For example, to configure an unrestricted service account with the slug `ci-build-account` and name `CI Build Account`, the resulting resource is: ```yaml service-account.yml icon="square-terminal" theme={null} api_version: v1/config kind: ServiceAccount spec: slug: ci-build-account name: CI Build Account unrestricted: true ``` 3. Create the resource by passing the file containing the resource to the `service-accounts create` command. ```shell theme={null} chronoctl service-accounts create -f service-account.yml ``` The command returns the service account resource, including its secret API `token`: ```yaml Service account resource icon="square-terminal" theme={null} api_version: v1/config kind: ServiceAccount spec: slug: ci-build-account name: CI Build Account token: acc261.... email: 7da....@serviceaccount.chronosphere.io unrestricted: true ``` This is the only time Observability Platform displays the service account's token. Store it securely. If you lose the token, you must delete and recreate the service account to generate a new token. To create an unrestricted service account with [Terraform](/tooling/infrastructure/terraform), use the `chronosphere_service_account` resource type and assign a `name` to use in Observability Platform inside the resource declaration. For example, this code creates a service account without any restrictions that Terraform refers to as `all_access`: ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_service_account" "all_access" { name = "Terraform - Unrestricted" } ``` 1. Add the definition to a Terraform file. 2. Run this command to create the resource: ```shell theme={null} terraform apply ``` Terraform stores the new service account's secret token in its state. To use the token in other Terraform resources, use the `.token` field of the service account resource. Using the previous example, the token for the service account with the Terraform resource name `all_access` is accessible at `chronosphere_service_account.all_access.token`. To view the token, inspect your Terraform state file. For example, you can use the [`jq`](https://stedolan.github.io/jq/) tool to filter service accounts in the state file: ```shell theme={null} jq '.resources[] | select(.type=="chronosphere_service_account")' terraform.tfstate ``` To complete this action with the Chronosphere API, use the [`CreateServiceAccount`](/tooling/api-info/definition/operations/CreateServiceAccount) endpoint and update the `service_account.unrestricted` key. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Create a restricted service account A restricted service account can access *only* the telemetry ingest and query APIs. The account can't access any of the other APIs. It can't make configuration changes, such as adding rules, creating monitors, or adding accounts. You can also optionally define **Label Restrictions** on restricted accounts to limit the account to querying or ingesting only specific labels or label-value pairs when authenticating with the account's token. The following types of restricted service accounts are available: * **Read-only** accounts are permitted only to query data. If you also define Label Restrictions, the account can query only metrics with the defined metric label keys or key-value pairs. * **Write-only** accounts are permitted only to ingest data. If you also define Label Restrictions, the account can ingest only metrics with the defined metric label keys or key-value pairs. * **Read & write** accounts are permitted to both query and ingest data, following any defined Label Restrictions. Write-only accounts are based on the principle of least privilege. A robot whose only job is to send ingest data to the server doesn't need query access. Restricted, read-only service accounts are restricted to reading even if the service account is added to a team with `Editor` or `SysAdmin` roles. Restricted accounts using label restrictions force a label with a specific value to be present on write, and allows queries that include only that label-value pair. Label restrictions support a maximum of two key-value pairs. Key-value pairs on individual service accounts must be unique. You can't set multiple values for the same label key. Observability Platform strips this label from query responses. Users with a restricted service account aren't aware that they're viewing metrics that match only the specified label. Users see all other telemetry. Select from the following methods to create a restricted service account. To create a restricted service account, you must have administrative privileges. 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Service Accounts**. 2. Click **+ Service Account**. This opens the **Add Service Account** dialog. 3. In the **Service Account Name** field, enter a descriptive name for the service account. 4. In the **Service Account Type** section, select **Restricted**. Observability Platform displays additional fields for you to complete. 5. Complete the creation process based on your needs. Use the **Label Restrictions** section to limit this account's metrics access to only labels with one, or a pair of specific key-value pairs. Click **+Add Label** to add a second key-value pair. Label restrictions are case sensitive. 6. Click **Save**. After creating the service account, Observability Platform displays its secret token. This is the only time Observability Platform displays the service account's token. Store it securely. If you lose the token, you must delete and recreate the service account to generate a new token. To create a restricted service account with [Chronoctl](/tooling/chronoctl), use the `service-accounts create` command. Specify required values as command arguments: * name (`--name`) * permission (`--permission`). Valid `permission` values are: * `READ`: Read-only access. * `WRITE`: Write access. * `READ_WRITE`: Read and write access. You can also specify optional metric label arguments: * label name (`--label-name`) * label value (`--label-value`) 1. Generate a templated service account resource with the `service-accounts scaffold` command: ```shell theme={null} chronoctl service-accounts scaffold > service-account.yaml ``` Replace `service-account.yaml` with any valid filename. 2. Edit the generated template to configure the new service account. Remove any fields that you don't need to configure. For example, to configure a restricted service account that can read and write only metrics with the label `tenant` and the value `name`, the resulting resource is: ```yaml service-account.yaml icon="square-terminal" theme={null} api_version: v1/config kind: ServiceAccount spec: slug: tenant-name name: tenant metrics_restriction: labels: tenant: name permission: READ_WRITE ``` 3. Create the service account: ```shell theme={null} chronoctl service-accounts create -f service-account.yml ``` The command returns the service account resource, including its secret API `token`: ```yaml Service account resource icon="square-terminal" theme={null} api_version: v1/config kind: ServiceAccount spec: slug: tenant-name name: tenant metrics_restriction: labels: tenant: name token: acc261.... email: 7da....@serviceaccount.chronosphere.io ``` This is the only time Observability Platform displays the service account's token. Store it securely. If you lose the token, you must delete and recreate the service account to generate a new token. To create a restricted service account using [Terraform](/tooling/infrastructure/terraform), use the `chronosphere_service_account` resource type followed by a resource name, and include a `restriction` section in the resource declaration. You must add a `permission` value in the `restriction` section to define the account's access. Valid `permission` values are: * `READ`: Read-only access. * `WRITE`: Write access. * `READ_AND_WRITE`: Read and write access. You can optionally add a `labels` value in the `restriction` section to additionally restrict the service account's access to limit metrics to display only those that have the specified label name and value. For example, this code creates a restricted service account that Terraform refers to as `restricted_labeled`, which can only read and write metrics with the label name `tenant` and label value `name`: ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_service_account" "restricted_labeled" { name = "Terraform - Restricted, READ_AND_WRITE, labeled" restriction { permission = "READ_AND_WRITE" labels = { "tenant" : "name" } } } ``` 1. Add the definition to a Terraform file. 2. Run this command to create the resource: ```shell theme={null} terraform apply ``` Terraform stores the new service account's secret token in its state. To use the token in other Terraform resources, use the `.token` field of the service account resource. Using the previous example, the token for the service account with the Terraform resource name `all_access` is accessible at `chronosphere_service_account.all_access.token`. To view the token, inspect your Terraform state file. For example, you can use the [`jq`](https://stedolan.github.io/jq/) tool to filter service accounts in the state file: ```shell theme={null} jq '.resources[] | select(.type=="chronosphere_service_account")' terraform.tfstate ``` To complete this action with the Chronosphere API, use the [`CreateServiceAccount`](/tooling/api-info/definition/operations/CreateServiceAccount) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. #### Label restriction example Label restrictions support a third-party use case, where Chronosphere is the first party, and a Chronosphere customer (CompanyX) is the second party. A Chronosphere customer's customer (CompanyA, CompanyB, or CompanyC) is the third party. In this scenario, CompanyX uses one tenant with Chronosphere to store data about CompanyA, CompanyB, and CompanyC, discriminated by some key (such as `third-party`) with a value of `A`, `B`, or `C`. Employees of CompanyX can see all of the data. Employees of CompanyA get a restricted service account with a label restriction of `third-party=A`. CompanyA doesn't know this key or this value. Whenever they send data to Chronosphere, Chronosphere augments the writes with `third-party=A`. Whenever CompanyA does queries, Chronosphere lets them see time series that include only `third-party=A`, but Chronosphere strips this label before showing the time series to employees of CompanyA. ## Delete a service account Select from the following methods to delete a service account. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To delete a service account, you must have administrative privileges. 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Service Accounts**. 2. Select the checkboxes for one or more service accounts you want to delete. 3. Click ** Delete** at the top of the list of service accounts. To delete a service account with [Chronoctl](/tooling/chronoctl), the account you use must either be a member of a team with the [`SysAdmin` role](/administer/accounts-teams/user-accounts#grant-system-administrator-sysadmin-privileges-to-users), or an unrestricted service account. Use the `service-accounts delete` command, passing the account's slug as the `--slug` argument. For example, to delete a service account with the slug `tenant-name`: ```shell theme={null} chronoctl service-accounts delete tenant-name ``` To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteServiceAccount`](/tooling/api-info/definition/operations/DeleteServiceAccount) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # Teams Source: https://docs.chronosphere.io/administer/accounts-teams/teams Chronosphere Observability Platform lets you create and manage teams, assign roles, and set permissions for effective collaboration. A *team* defines how an organization grants permissions for sensitive management and administrative operations to *accounts*, including [*user accounts*](/administer/accounts-teams/user-accounts) and [*service accounts*](/administer/accounts-teams/service-accounts). A team's accounts have a shared responsibility for certain services. Chronosphere Observability Platform represents those services as [*collections*](/administer/collections), which contain resources related to the service a collection represents, such as dashboards and monitors. A team can own multiple collections. All user and service accounts in a team that has the special System Administrator (`SysAdmin`) role are *administrators*. If you're an administrator, you can [add this role to a team](#add-a-role-to-a-team). Teams can own collections and services, which in turn own other functions: ```mermaid actions={false} theme={null} flowchart TD accDescr: Ownership flow for teams and functions Team[**Team**] Col(Collections) Ser(Services ) Team --> Col Team --> Ser Dash(Dashboards) Mon(Monitors) SLO(SLOs) Col --> Dash Col --> Mon Col --> SLO Dash2(Dashboards) Mon2(Monitors) SLO2(SLOs) Ser --> Dash2 Ser --> Mon2 Ser --> SLO2 ``` ## View teams To search for a team: 1. Click the search bar to focus on it, or use the keyboard shortcut `Control+K` (`Command+K` on macOS). 2. Begin typing any part of the team's name. 3. Optional: Click the **Teams** filter at the top of the search results to constrain your results to only Teams. 4. Click the search result you're interested in, or use the arrow keys to select it and press enter, to go to that team. ### View teams and their collections To view teams and the collections they own, in the navigation menu exit the **Admin** menu (if needed, click ** Exit Admin**) and select ** Collections**. For details about working with collections, see [Collections](/administer/collections). ### View a list of teams to manage To view teams in a list that displays their members, provides an audit log of membership changes, and includes management options for administrators: In the navigation menu, click ** Go to Admin** and then select **Platform > Teams**. Use the command: ```shell theme={null} chronoctl teams list ``` For details, see [Accounts and Teams](/administer/accounts-teams#view-accounts-and-teams). ## Manage teams To manage teams, you must use a user account or [service account](/administer/accounts-teams/service-accounts) that belongs to a team with the SysAdmin role. To manage teams, in the navigation menu, click ** Go to Admin** and then select **Platform > Teams**. ### Create a team To create a new team: 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Teams**. 2. Click **Create team**. 3. In the dialog that appears, enter a name for the new team and the access role it should have. 4. Click **Save** to create the team. The team now appears in the list of users and teams. To create teams or modify team membership with Chronoctl, you must use a user account or [service account](/administer/accounts-teams/service-accounts) that belongs to a team with the SysAdmin role. To create and manage a team with [Chronoctl](/tooling/chronoctl), use the `Team` resource type. The resource can contain these fields: | Field | Type | Description | | ------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `name` | String | The team name as displayed in Observability Platform. | | `slug` | String | Optional: A unique identifier that Observability Platform uses to identify the team. If omitted, Observability Platform generates a slug. | | `description` | String | Optional: A description displayed when viewing this team. | | `user_emails` | Collection of strings | Contains an item for each member account's email address. | To generate an example Team resource, run `chronoctl teams scaffold`. If an email address in `users` doesn't already have an Observability Platform account, the app creates a pending user account for that email address. When that user signs in, Observability Platform then activates the account. For example, this YAML resource creates a team named `DBAs` in Observability Platform, with the description `Database administrators`, and which the app identifies with the slug `dbas`. It also includes the listed accounts identified by their email addresses as team members: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Team spec: name: DBAs slug: dbas description: "Database administrators" user_emails: - adam@example.com - betty@example.com ``` To include service accounts, add the service account's internal email address to the `users` list. For details, see [Service Accounts](/administer/accounts-teams/service-accounts). To create teams or modify team membership with Terraform, you must use a user account or [service account](/administer/accounts-teams/service-accounts) that belongs to a team with the SysAdmin role. To create and manage a team with [Terraform](/tooling/infrastructure/terraform), declare the `chronosphere_team` resource type and provide a resource name. The resource can contain these fields: | Field | Type | Description | | ------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `name` | String | The team name as displayed in Observability Platform. | | `slug` | String | Optional: A unique identifier that Observability Platform uses to identify the team. If omitted, Observability Platform generates a slug. | | `description` | String | Optional: A description displayed when viewing this team. | | `user_emails` | Array of strings | The email address for each member account. | For example, this resource creates a team referred to by Terraform as `dbas`, named `DBAs` in Observability Platform, and identified in the app with the slug `dbas`. It also includes the listed accounts identified by their email addresses as team members: ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_team" "dbas" { name = "DBAs" slug = "dbas" description = "Database administrators" user_emails = [ "adam@example.com", "betty@example.com", ] } ``` To include service accounts, add the service account's internal email address to the `user_emails` array. For details, see [Service Accounts](/administer/accounts-teams/service-accounts). 1. Add the definition to a Terraform file. 2. Run this command to create the resource: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`CreateTeam`](/tooling/api-info/definition/operations/CreateTeam) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Edit team membership To add users to or remove users from a team: 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Teams**. 2. [Select the team](/administer/accounts-teams#view-accounts-and-teams) to manage. 3. Click ** Edit Team**. 4. In the **Team Members** section, click **Update Membership**. 5. From the account list that appears, select the checkbox for each account you want to add to the team, and clear the checkbox for each account you want to remove from the team. 6. Click **Save** to edit the team's membership. The team page updates its membership list accordingly, and the audit log adds an entry logging the operation. The account list includes both user and service accounts. You can remove a service account from the **Default** team to make it read-only. For more information, see [Assign team roles and permissions](#assign-team-roles-and-permissions). To modify teams with Chronoctl, you must use a user account or [service account](/administer/accounts-teams/service-accounts) that belongs to a team with the `SysAdmin` role. You can use [Chronoctl](/tooling/chronoctl) to replace and update a team and its membership with those defined in a YAML file. To generate a YAML description of your current teams and accounts, see [View Accounts and Teams](/administer/accounts-teams#view-accounts-and-teams). Following the `DBAs` team example from [creating a team](#create-a-team), to add a user to that team, update the YAML to include the new user's email: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Team spec: name: DBAs slug: dbas description: "Database administrators" user_emails: - adam@example.com - betty@example.com - charles@example.com ``` To apply the changes, use the Chronoctl `chronoctl teams update` command, which updates teams but doesn't create them if they don't already exist: ```shell /FILE_NAME/ theme={null} chronoctl teams update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the path to the YAML definition file. You can test the changes without applying them by including the `--dry-run` flag: ```shell /FILE_NAME/ theme={null} chronoctl teams update -f FILE_NAME.yaml --dry-run ``` To force `teams update` to create a defined team if it doesn't exist, include the `--create-if-missing` flag: ```shell /FILE_NAME/ theme={null} chronoctl teams update -f FILE_NAME.yaml --create-if-missing ``` To edit a team's membership using [Terraform](/tooling/infrastructure/terraform), modify the `user_emails` array in the team's `chronosphere_team` resource, and then apply the resource using Terraform. To complete this action with the Chronosphere API, use the [`UpdateTeam`](/tooling/api-info/definition/operations/UpdateTeam) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Assign team roles and permissions Observability Platform teams use *roles* to grant or restrict access to features that create, edit, or delete resources, teams, and accounts. Roles are assigned to teams, and all accounts that belong to a team gain the permissions granted by that team's role. ### Default team properties During creation, accounts are assigned to the **Default** team, which has the `Editor` role. The **Default** team can't be deleted, and its role can't be changed. To remove editing access from a user or service account, [remove them from the **Default** team](#edit-team-membership). This limits the account to read-only Viewer access if they don't have edit access through membership in another team. [Unrestricted service accounts](/administer/accounts-teams/service-accounts#create-an-unrestricted-service-account) removed from the **Default** team have read-only access to the full Chronosphere API. To create a read-only unrestricted service account that can access all APIs, remove it from the **Default** team. ### Available roles The available roles are: * `None`: Accounts on teams with this role can view pages in Observability Platform but can't create, edit, or remove resources or accounts. Any [Assistant](/navigate/assistant) chats by these accounts aren't saved to the chat history. Accounts that don't belong to any team also have the `None` role by default. * `Editor`: Accounts on teams with this role can create, view, edit, and delete resources within Observability Platform, but can't create, edit, or delete accounts and teams. The Assistant saves the last 20 chat sessions of accounts that have the `Editor` or higher roles. * `SysAdmin`: Any account that's a member of a team with the System Administrator (`SysAdmin`) role can create, edit, and delete accounts and teams, assign accounts to teams, and assign roles to teams. Accounts with this role are designated by a key icon next to their name in the **Users** list. * `User Administrator`: Accounts belonging to teams with this role can access the System for Cross-domain Identity Management (SCIM) API, which manages accounts that use single sign-on. This role grants no other permissions. The `User Administrator` role can be assigned to teams only in organizations that have SCIM integration enabled. For more information, see [Okta user synchronization](/administer/accounts-teams/okta). If an account belongs to multiple teams with different roles, they inherit the highest-level permissions of each assigned team. For example, an account in a team with the `Editor` role and a second team with the `SysAdmin` role has both sets of privileges. The following chart summarizes roles and permissions: | Action | None | Editor | SysAdmin | User Admin | | -------------------------------------------------- | :--------------------------: | :--------------------------: | :--------------------------: | :--------------------------: | | Interact with SCIM API | | | | | | View pages in Observability Platform | | | | | | Add and edit user and service accounts | | | | | | Add, edit, and remove teams | | | | | | Change users' team membership | | | | | | Apply service configuration changes | | | | | | Add, edit, and remove trace control plane datasets | | | | | | Add, edit, and remove derived labels | | | | | | Add, edit, and remove other resources | | | | | Some features vary by role beyond whether an account can use them. Accounts with the `None` role can chat with the [Assistant](/navigate/assistant), but their conversations aren't saved, don't appear in chat history, and can't be shared. See [Availability and permissions](/navigate/assistant#availability-and-permissions). ### Add a role to a team To add a role to a team: 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Teams**. 2. [Select the team](/administer/accounts-teams#view-accounts-and-teams) to manage. 3. In the **Team Information** section, click ** Edit**. 4. Click the **Role** dropdown and select the role you want to assign to the team. 5. Click **Save**. ### Remove a role from a team To remove a role from a team: 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Teams**. 2. [Select the team](/administer/accounts-teams#view-accounts-and-teams) to manage. 3. In the **Team Information** section, click ** Edit**. 4. Click the **Role** dropdown and select the **None** role. 5. Click **Save**. ### Rename a team To rename a team: 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Teams**. 2. [Select the team](/administer/accounts-teams#view-accounts-and-teams) to manage. 3. In the **Team Information** section, click ** Edit**. 4. Change the team's name in the **Name** field. 5. Click **Save**. ### Describe a team Team pages can display a brief description in the **Team Information** section. You can use this description to clarify the team's purpose, identify the people responsible for managing it, or provide general information or context to the team's activities. You can also add links to URLs, whether within or external to Observability Platform, to the **Team Information** section. To access a team's information: 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Teams**. 2. [Select the team](/administer/accounts-teams#view-accounts-and-teams) to manage. 3. In the **Team Information** section, click ** Edit**. To add, edit, or remove a team's description: 1. Enter the team description in the **Description** field. 2. Click **Save**. To add links to a team's information: 1. In the **Team Links** section, click ** Add**. This adds a row of fields to the section. 2. Enter the link text in the **Link Name** field. 3. Enter the link target in the **URL** field. 4. Repeat these steps to add more links. 5. Click **Save**. To edit a team's links: 1. In the **Team Links** section, modify the link's **Link Name** and **URL** fields. 2. Click **Save**. To remove links from a team's information: 1. In the **Team Links** section, click the delete button in the rows of the links you want to delete. 2. Click **Save**. ### Delete a team Use one of the following methods to delete a team. The **Default** team can't be deleted. To delete a team: 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Teams**. 2. [Select the team](/administer/accounts-teams#view-accounts-and-teams) to delete. 3. In the **Team Information** section, click ** Edit**. 4. Click **Delete team**. 5. In the confirmation dialog that appears, click **Delete**. To delete teams with [Chronoctl](/tooling/chronoctl), use the `chronoctl teams delete` command: ```shell /SLUG/ theme={null} chronoctl teams delete SLUG ``` Replace *`SLUG`* with the unique slug of the team you want to delete. For example, to delete the team with the slug `dbas`, run: ```shell theme={null} chronoctl teams delete dbas ``` To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteTeam`](/tooling/api-info/definition/operations/DeleteTeam) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # User accounts Source: https://docs.chronosphere.io/administer/accounts-teams/user-accounts Chronosphere Observability Platform lets you manage user access by handling account creation, modification, and role assignments. A *user account* represents a user's identity in Chronosphere Observability Platform. Accounts can belong to [*teams*](/administer/accounts-teams/teams), which group and distribute permissions to its members. ## Grant System Administrator (`SysAdmin`) privileges to users You can grant permissions to teams, which then confer those permissions to its users. A user account that's a member of a team with the `SysAdmin` role has administrative access to Observability Platform features. This includes the abilities to create teams and user accounts, assign users to teams, and define team permissions. For instructions about assigning roles to a team, see [Add a role to a team](/administer/accounts-teams/teams#add-a-role-to-a-team). ## Authenticating as a user To [authenticate](/overview/authenticating) with Observability Platform, a user typically signs in interactively with their user account. A user account can also use a temporary [personal access token](/administer/accounts-teams/personal-access-tokens) for non-interactive authentication, such as with tools like Chronoctl or for clients that interact with the Chronosphere API. Observability Platform attributes all actions that a user takes to their user account. User accounts are distinct from *service accounts*, which provide identities for services and allow administrators to define what each service can access. Service accounts authenticate non-interactively using a unique API token permanently associated with each account. For details, refer to [Service accounts](/administer/accounts-teams/service-accounts). ## View accounts You can view accounts in the navigation menu, or by using Chronoctl. In the navigation menu, click ** Go to Admin** and then select **Platform > Users**. For details, see [Accounts and teams](/administer/accounts-teams#view-accounts-and-teams). ## Add a user account To add user accounts, you must use an account that belongs to a team with the `SysAdmin` role. This process applies only to accounts *without* single sign-on enabled. For details, see [Authenticating with Chronosphere](/overview/authenticating). To add a user account, you must have administrative privileges: 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Users**. 2. Click ** Add user**. 3. Enter the user's email address into the field. 4. Click **Invite User**. 5. Optional: Add the user to a [team](/administer/accounts-teams/teams). Users without an assigned team receive viewer permissions. Observability Platform sends an invitation email to the address containing a link to verify and access the user account. After the user accepts the invitation, they must verify their account, at which point the user's email address appears in the list of accounts with a green checkmark. ## Delete a user account To remove a user account from Observability Platform: 1. If your environment uses a single sign-on (SSO) provider, remove the user account from the identity provider (IdP) permission group to prevent the user from signing in to Observability Platform. 2. To remove the user account from display in Observability Platform itself, create a ticket with Chronosphere Support and request to have the user account deleted. ### Users of Okta with SCIM If you're using Okta and have SCIM integrated with the Observability Platform connection, removing the user from access in the IdP deprovisions and removes the user from Observability Platform. ### Service accounts Ensure a service account isn't being used before you delete it. Service account tokens are used by critical components of Observability Platform (including the Chronosphere Collector), and incorrectly deleting a service account can significantly impact your environment. Deleted user accounts with access to a service account token can continue to access Observability Platform when using tools like Chronoctl and Terraform. To avoid access by these accounts, delete any service accounts created by deleted user accounts. To find these accounts, in the navigation menu, click ** Go to Admin** and then select **Platform > Service Accounts** and review the **Created By** column. # Audit logs Source: https://docs.chronosphere.io/administer/audit-logs View recent system and user actions in the Audit Log, including event details and the accounts and apps that generated them. To view recent system and user actions, view the Audit Log, which is available to users with administrative privileges. In the navigation menu, click ** Go to Admin** and then select **Platform > Audit logs**. Click one of the listed actions or events to view its details, including the following: * Description of the event. * Date and time of the event. * Service account ID of the user who performed the event. * App that generated the event. * App method that generated the event * Method resource that generated the event. * Full JSON payload of the event. # Collections Source: https://docs.chronosphere.io/administer/collections Learn how collections group dashboards, monitors, and other resources so a team can manage a service at scale. A *collection* is a group of Chronosphere Observability Platform resources, such as [dashboards](/observe/dashboards) and [monitors](/investigate/alerts/monitors), associated with a service or feature you want to monitor. A [*team*](/administer/accounts-teams/teams), consisting of [*user accounts*](/administer/accounts-teams/user-accounts) and [*service accounts*](/administer/accounts-teams/service-accounts) that share responsibility for the service or feature, is also responsible for each collection. Collections help service and feature owners manage their services at scale and streamline incident response by consolidating all relevant information about a service or feature into one place, and sharing access to that information with everyone in its responsible team. For example, to ensure that your team and on-call engineers all have full context for a service, you can create a collection for it that provides direct access to its monitors, dashboards, and the team members who can respond to incidents. Observability Platform then provides a menu of a team's collections to its members who can quickly review resources most relevant to their services, and a homepage for the collection that the team can share with on-call engineers or other stakeholders. ## View collections To search for a collection: 1. Click the search bar to focus on it, or use the keyboard shortcut `Control+K` (`Command+K` on macOS). 1. Begin typing any part of the collection's name. 2. Optional: Click the filters for all other listed resource types at the top of the search results to remove them and display only collections. 3. Click the search result you're interested in, or use the arrow keys to select it and press `Enter` (`Return` on macOS), to select that collection. To browse collections by their respective teams, in the navigation menu click ** Collections**. Collections associated with teams that you belong to appear first, followed by lists of **All Teams & Collections** and **Unowned Collections**. To expand the latter lists, click the list's expand icon. An *unowned collection* is a collection that isn't assigned to a team. Unowned collections often originate from *buckets*, an earlier container for organizing resources that Observability Platform automatically converted to unowned collections. To resolve an unowned collection, [assign it to a team](#edit-a-collection). If you have administrative privileges, you can view a filterable list. 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Collections**. Clicking a collection or team in this table takes you to their respective homepages. * To filter this table by keyword, click **Search collections** and begin typing. The table automatically updates to display only the matching collections. * To filter this list by teams, select them in the **Select a Team** dropdown. * To display only collections that lack an owner, enable the **Unowned Collections Only** toggle. You can optionally select collections by checking the checkbox on each collection's row, which reveals an **Edit Team Association** button. Clicking this button allows you to select a parent team for all selected collections. If you select a team, click **Save** to save the edited association. For more Collection editing features, see [Edit a Collection](#edit-a-collection). To view a list of all collections using [Chronoctl](/tooling/chronoctl), use the `collections list` command: ```shell theme={null} chronoctl collections list ``` This returns a list of YAML documents separated by three dashes (`---`), each representing a collection: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Collection spec: slug: example-collection name: Example Collection team_slug: example-team description: Example Team - Example Collection ``` To complete this action with the Chronosphere API, use the [`ListCollections`](/tooling/api-info/definition/operations/ListCollections) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create a collection You can use Observability Platform to create Terraform, Chronoctl, or API configurations, which you must then apply using the respective application. Collections use a `slug` as an identifier. [Services](/observe/services) also use a `slug`. Collection and service slugs can have the same value. To create a collection, you must have administrative privileges. 1. In the navigation menu, click ** Go to Admin** and then click **Platform > Collections**. 2. Click **Create collection**. 3. In the dialog that appears, enter a descriptive name in the **Collection Name** field. Observability Platform displays this name anywhere that it refers to this collection. 4. Optional: Click the **Parent team** dropdown and select a team to which you're associating this collection. If the dropdown doesn't contain any teams, you can create one. 5. Optional: Enter a description in the **Collection Description** field. Observability Platform displays this description when you view this collection's homepage. 6. Click **Save**. To create a collection using [Chronoctl](/tooling/chronoctl), apply a `Collection` resource. The resource contains these fields: | Field | Type | Description | | -------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | String | The collection name as displayed in Observability Platform. | | `slug` | String | Optional: A unique identifier that Observability Platform uses to identify the collection. If omitted, Observability Platform generates a slug. | | `team_slug` | String | Optional: The slug of the team with which you're associating with this collection. | | `description` | String | A description displayed when viewing this collection. | | `notification_policy_slug` | String | The slug of a notification policy that you're associating with this collection. | For example, this resource creates a new collection named `DBAs` with the slug `dbas-collection` and associated with `dbas-team`: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Collection spec: slug: dbas-collection name: DBAs team_slug: dbas-team description: Collection of database-related resources. notification_policy_slug: dbas-policy ``` To create the collection: 1. Save the `Collection` resource definition to a file, such as `collection.yaml`. 2. Test the changes with a dry run: ```shell theme={null} chronoctl collections create -f collection.yaml -d ``` 3. Apply the changes: ```shell theme={null} chronoctl collections create -f collection.yaml ``` To create a collection using [Terraform](/tooling/infrastructure/terraform), use the `chronosphere_collection` resource type. | Field | Type | Description | | ------------------------ | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | String | The collection name as displayed in Observability Platform. | | `slug` | String | Optional: A unique identifier that Observability Platform uses to identify the collection. If omitted, Observability Platform generates a slug. | | `team_id` | String | Optional: The identifier of the team with which you're associating with this collection. | | `description` | String | A description displayed when viewing this collection. | | `notification_policy_id` | String | Optional: The identifier of a notification policy that you're associating with this collection. | For example, this resource creates a new collection that Terraform refers to as `databases`, names it `Databases` in Observability Platform, provides a description, associates the collection with a notification policy, and assigns the collection to the team `dbas`: ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_collection" "infra" { name = "Databases" team_id = chronosphere_team.dbas.id description = "Collection of resources related to database services." notification_policy_id = chronosphere_notification_policy.np.id } ``` To complete this action with the Chronosphere API, use the [`CreateCollection`](/tooling/api-info/definition/operations/CreateCollection) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Edit a collection Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To edit a collection, you must have administrative privileges. 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Collections**. 2. Click the name of the collection you want to modify. 3. In the **Collection information** section, click ** Edit**. 4. In the dialog that appears, edit the collection based on your needs. 5. Click **Save**. You can optionally select collections by checking the checkbox on each collection's row, which reveals an **Edit Team Association** button. Clicking this button allows you to select a parent team for all selected collections. If you select a team, click **Save** to save the edited association. For more Collection editing features, see [Edit a Collection](#edit-a-collection). To edit a collection using [Chronoctl](/tooling/chronoctl), modify an existing `Collection` resource, then apply the change. To view `Collection` resources as Chronoctl YAML, see [View collections](#view-collections). 1. Copy the `Collection` resource for the collection you want to edit. 2. Save the collection to a file, such as `collection.yml`. 3. Modify the collection's fields. For details, see [Create a collection](#create-a-collection). 4. Test the changes with a dry run: ```shell theme={null} chronoctl collections create -f collection.yaml -d ``` 5. Apply the changes: ```shell theme={null} chronoctl collections create -f collection.yaml ``` To edit a collection using [Terraform](/tooling/infrastructure/terraform), modify its resource and apply it using Terraform. To complete this action with the Chronosphere API, use the [`UpdateCollection`](/tooling/api-info/definition/operations/UpdateCollection) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete a collection Before you delete a collection, you must first remove all of its monitors and dashboards. You can do this by either deleting them or moving them to other collections. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To delete a collection, you must have administrative privileges. 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Collections**. 2. Click the name of the collection you want to delete. 3. In the **Collection information** section, click ** Edit**. 4. In the dialog that appears, click **Delete collection**. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To delete a collection using [Chronoctl](/tooling/chronoctl), use the `collections delete` command: ```shell /SLUG/ theme={null} chronoctl collections delete SLUG ``` Replace *`SLUG`* with the slug for the collection you want to delete. For example, to delete a collection with the slug `dbas-collection`, run: ```shell theme={null} chronoctl collections delete dbas-collection ``` To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteCollection`](/tooling/api-info/definition/operations/DeleteCollection) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # Team and collection home pages Source: https://docs.chronosphere.io/administer/collections/home View team and collection home pages to see associated monitors, alert statuses, dashboards, and context. Each [*team*](/administer/accounts-teams/teams) and [*collection*](/administer/collections) provides a *home* page that lists its associated resources. These pages provide an overview of alerting [monitors](/investigate/alerts/monitors) and their statuses, links to associated [dashboards](/observe/dashboards), and additional context about the team or collection. ## View a home page To go to a team's or collection's home page: 1. In the navigation menu exit the **Admin** menu (if needed, click ** Exit Admin**) and select ** Collections**. Chronosphere Observability Platform first lists collections associated with teams to which you belong under **My Teams & Collections**, then under **All Teams & Collections** lists each team and the collections it's responsible for. 2. Click a team or collection to view its home page. ### Alerting monitors The **Monitors** section includes a table that lists any [monitors](/investigate/alerts/monitors) associated with this team or collection that are reporting alerts. The table's columns provide pertinent details and links to each monitor: * **Status:** The table lists any monitors associated with the team or collection that have active alerts, sorted by descending severity from critical to warning levels. Monitors not reporting alerts, including fully muted monitors, aren't listed here. To view all monitors associated with this team or collection, click the section's **View all monitors** link. * **Name:** Each alerting monitor's name is a link to its **Monitor Info** view. * **Alerting For:** This indicates the duration since the monitor triggered the alert. * **Notification Policy:** When viewing a collection, each alerting monitor links to its [notification policy](/investigate/alerts/notifications/policies). * **Collection:** When viewing a team, each alerting monitor links to its associated collection home page. #### Create a monitor To create a monitor from a team's or collection's homepage, click the **+ Add** button in the homepage's Monitors section. This opens the **Add Monitor** panel. For more information, see [Create a monitor](/investigate/alerts/monitors#create-a-monitor). ### Dashboards The **Dashboards** section links to the first 10 dashboards associated with this team or collection in alphabetical order. To view all associated dashboards, click **View all dashboards**. Team home pages exclusively show links to each dashboard's associated collection. #### Create a dashboard To create a dashboard from a team's or collection's homepage, click the **+ Add** button in the homepage's Dashboards section. This opens the **Add Dashboard** modal. For more information, see [Create a dashboard](/observe/dashboards#create-a-new-dashboard). ### Manage connected resources You can connect collections to monitors and dashboards which are owned by another collection or service. To connect a monitor or dashboard to a service from a collection's home page: 1. Next to **Monitors** or **Dashboards**, click **Manage**. 2. Select **Connections**. 3. Add or remove a [connected resource](/observe/services/extending-services#view-connected-resources). ### Collections Team home pages have a **Collections** section that links to each collection associated with the team, and includes text from the collection's description if available. #### Create a collection To create a collection from a team's homepage, click the **+ Add** button in the homepage's Collections section. This opens the **Add Collection** modal. For more information, see [Create a collection](/administer/collections#create-a-collection). ### Notification policies Team home pages have a **Notification Policies** section that links to each notification policy associated with the team. ### Links The **Links** section contains links added to a team or collection home. These links can point to any `http` or `https` URL. ### Description Teams and collections can have a text description that's displayed in the **Description** section of their home pages. This description can also appear in other lists that include teams or collections, or when listing them in [Chronoctl](/tooling/chronoctl). ### Team members A team's home page also lists all team members sorted by their [account's](/administer/accounts-teams) email address. The list indicates accounts with administrative access by listing **Admin** with their email address. Service accounts have an email address consisting of its slug followed by `@serviceaccount.chronosphere.io`. ## Edit a home page Each home page generates its content based on its team's or collection's associations with other resources. To update those associations, see the documentation for those resources. ### Teams To edit a team's name, description, or membership, see [Teams](/administer/accounts-teams/teams). Editing these details requires [administrative access](/administer/accounts-teams/user-accounts#grant-system-administrator-sysadmin-privileges-to-users). ### Collections To edit a collection's name, description, or team association, see [Collections](/administer/collections). Although you can click the ** Edit Collection** button in a collection's home page and view its name and description, you must use Terraform to update these details. ### Links To add links to the **Links** section: 1. Go to the team's or collection's home page. 2. Click ** Edit Team** or ** Edit Collection**. 3. Under **Team Links** or **Collection Links**, click **+ Add**. 4. Enter a **Link Name** and **URL**. 5. Click **Save**. To remove links from the **Links** section: 1. Go to the team's or collection's home page. 2. Click ** Edit Team** or ** Edit Collection**. 3. Under **Team Links** or **Collection Links**, click **+ Add**. 4. Click the delete button next to the link you want to delete. 5. Click **Save**. # Design service level objectives Source: https://docs.chronosphere.io/administer/design-slos Learn about fundamental concepts and best practices when designing service level objectives (SLOs) in Chronosphere Observability Platform. Effective service level objectives (SLOs) can take time and research to create. Although Observability Platform's [SLOs](/observe/slo) are built around industry best practices, you can tailor your SLOs to make them more effective alerting and observation tools for your individual services. ## Design user-focused indicators SLOs should measure availability of your services from your users' perspective. Design your SLOs to identify when services are falling short of your users' needs. Availability isn't always a binary state of up or down. Define your service level indicators (SLIs) with your users' experience in mind. Slow responses, non-blocking errors, or unexpected results can represent a lack of service availability from your users' perspective, even if your service is technically available and responsive. Observability Platform SLOs are dynamic and can provide multiple error budgets from a single query. Leverage these features when designing your indicators to create low-maintenance SLOs that are also focused on metrics relevant to your users' experience. Some services might require tracking multiple SLI definitions, such as tracking both latency and availability. In such situations, each SLI should have its own SLO page with its own burn rate and alerting configuration. ## Set a reasonable objective Although it might seem ideal to set a perfect target of 100% availability as your objective, SLOs are most effective when they recognize that issues are inevitable. Instead of aiming for perfection, define your objectives around your users' tolerance for failures to meet their expectations. This tolerance is inherently subjective. Beyond minimums set in legal agreements and SLAs, you can iterate on your SLOs based on user feedback and research, your development pace, and your ability to absorb risk. Likewise, the error budgets created by your objectives help you define the amount of risk you're willing to accept for a given service. This in turn helps you [plan risky actions](#design-slos-for-risk-management), such as potentially disruptive deployments, around your users' tolerance for downtime. If possible, define your objective based on historical performance to ensure your targets are realistic, and also to minimize on-call burdens on your responders. ## Determine an appropriate unit to measure * Error ratio objectives help you identify issues with services where you have a low tolerance for any number of errors. * Time slice objectives help you identify issues with services where the length of an incident is more relevant than the total number of errors, and can reduce the noise of transient or low-impact errors. Many SLOs measure the ratio of errors to total measurements over a time window. Observability Platform refers to these as *error ratio SLO*. The resulting percentile provides a straightforward indicator of the measured service's health over time. Its error budget also refers to the ratio of errors that can be tolerated over the remaining time window before the objective is breached. Error ratio SLOs can be valuable when your service has a low tolerance for errors of any type, regardless of how long they degrade the service's performance. Since all errors count against the error budget in an error ratio objective, you can track patterns of error counts over time to identify periodic or intermittent errors before they degrade your service's availability. Burn rate measurements can also alert you to spikes in errors at the early stages of an incident. However, total error counts might not accurately reflect a service's availability. The amount of time during which the service's performance was degraded can matter more to end users than the total number of recorded errors. For such services, use a *time slice SLO*, which instead measures intervals within the time window to determine how long a service was degraded. In a time slice SLO, the indicator and error budget refer to the percentage of time during the time window that the service was available or degraded. Instead of a certain number of errors triggering an objective's breach, a time slice SLO is breached when the system is degraded for a percentage of time during the window that exceeds the objective. Time slice SLOs use intervals as small as one to five minutes. Choose the interval based on your service's behavior when degraded and its affects on the service's users. Since each slice is calculated independently, the objective only needs to aggregate data for each time slice, instead of across the entire time window, which can be weeks in length. Services that can benefit from time slice SLOs might experience relatively uniform load over the time window, don't have scheduled or expected downtime or outages, and can safely recover from intermittent errors. However, these traits can mask occurrences of low-impact and intermittent errors that still occur but fail to breach the threshold of each time slice. Time slice SLOs can also delay responses to incidents and burn rate measurements, especially over longer time slice intervals, since the success or failure of a slice can be determined only when it breaches the slice's threshold. ## Use template variables to reduce query maintenance If you write your SLO's query, use [template variables](/investigate/alerts/manage-slos) to refer to your time window (`{{.Window}}`) or time slice interval (`{{.TimeSlice}}`), dimensions (`{{.GroupBy}}`), and label filters (`{{.AdditionalFilters}}`). These variables automatically align your query to SLO changes, and also help facilitate configuration as code by single-sourcing their definitions. ## Tune time window and burn rate definitions Observability Platform uses opinionated default time windows and multi-window burn rates, all based on industry best practices. If you intend to change time window and burn rate definitions, ensure that they remain realistic and stay mindful of the alerting noise that might result from changes. When redefining time windows and burn rates, consider the following: * Prioritize alerts: Ensure that alerts are prioritized by severity so that higher burn rates trigger more urgent action than long-term trends or slower burn rates. * Mind services' traffic volume: Services with lower traffic levels might have inconsistent or spiky error rates that cause false positives. Use windows with longer time frames and more conservative burn rates to reduce noise in your alerts. ## Design SLOs for rapid response to issues An SLI measures your service's error rates across a defined time window to determine whether your service achieves its objective. The SLO also provides tools that help responders protect your service from breaching its objective. Burn rates measure your error rates in time windows as small as several minutes, rather than days or weeks. Burn rates can trigger alerts on the implication that if a high error rate across a short time span continues unabated, then your SLO will breach its objective before the end of its time window. Observability Platform's defaults provide multiple burn rates. SLOs provide measurements across multiple windows per burn rate to reduce false positives. By setting burn rate alerts, your SLO can identify and alert responders when a service rapidly experiences more errors or downtime than expected. Your responders can then intervene long before the error budget is exhausted. ## Design SLOs for risk management You can also use SLOs in risk management and planning. Error budgets are designed to be spent, and you can use them to plan upcoming deployments that you know might deplete them. For example, downtime from planned deployments and maintenance activities are part of your error budget, and burn rate alerting can help you identify and react when such planned actions have unexpected user-facing results. Consider your error budget separately from your SLO objective. If you set a 99% objective, consider your 1% error budget as its own amount of capacity that you can spend on risky deployment or maintenance actions. Burn rates measure consumption of your error budget rather than your total objective because they extrapolate how much capacity you can sacrifice before your service breaches its objective. Burn rate alerts help responders react to issues as they happen, and also help identify how much downtime your users can tolerate for the rest of your time window. An incident with a high burn rate leaves less error budget for the rest of your time window, which affects how you allocate the remainder. Conversely, reducing the downtime of risky actions gives you more budget to work with for more frequent or riskier actions within your time window. Use burn rate alerts to also alert stakeholders who determine deployment schedules, and use visualizations in an SLO's page to find historical context when planning deployments for future time windows. ## Create effective SLO alerts For managing and responding to degraded service performance and outages, SLOs provide significant benefits compared to other alerting practices: * User-centric measurement: SLOs focus on visualizing and reporting on symptoms rather than causes, which concentrates coverage on issues actively affecting your services and reduces false positives. * Standardized operational practices: The standardized features and presentation of SLOs facilitate normalized alerts, dashboards, and operational reviews across your organization to improve consistency in team transitions and on-call rotations. * Data-driven decision making: By measuring error budgets against availability targets, SLOs provide objective data toward balancing investments in a service's reliability against new feature development. This allows for more consistent risk management while you iterate on the service's implementation. When you [define your SLO](/investigate/alerts/manage-slos#define-an-slo), use the **SLO** tab in the **SLO preview** drawer to simulate alerts. This tab uses real data to project where your SLO would have triggered alerts, and you can update those simulations after tuning your objective and burn rates. ### Avoid high-impact alerts on new SLOs New SLOs often require some iteration and tuning to become effective alerting tools. The best-designed objectives and alerts can still result in alerts triggering too quickly or too often. For new SLOs, create alerts with a trial period of a few weeks. Use lower-impact notification policies during this period to avoid recurring alerts, and use this period to tune your SLO's objective, burn rates, and alerting settings. After you've ensured that the SLO alerts your responders only when necessary, switch your SLO to a higher-impact notification policy. ## Use SLOs with other Observability Platform features In addition to alerts, Observability Platform SLO integrate with other features that help you identify, analyze, and investigate issues. * Use [Differential Diagnosis (DDx) for metrics](/investigate/analyze/differential-diagnosis/metrics) from SLO visualization panels to help identify the source of spikes or other unusual shapes. * Connect SLOs to [services](/observe/services), which includes the SLO's status with other monitors when depicting the service's health. This can draw responders' attention to SLOs when viewing a [service page](/observe/services/service-pages). ## Further reading SLOs are a complex subject, and resources from across the observability industry can help you better understand them and improve your SLO designs. * [SRE Fundamentals: SLA versus SLO versus SLI](https://chronosphere.io/learn/know-the-sre-fundamentals-differences-between-sli-vs-slo-vs-sla/) in Chronosphere's Resource Center provides a high-level overview of SLO components, purpose, and terminology. * [The Art of SLOs](https://sre.google/resources/practices-and-processes/art-of-slos/) workshop by Google's SRE team provides a theoretical basis and practical hands-on examples of effective indicators and objectives. # Licensing and system limits Source: https://docs.chronosphere.io/administer/limits-licensing Learn about licensing and system limits and how they differ. Chronosphere Observability Platform enforces two categories of limits that govern how telemetry data flows through the system: license limits and system limits. To stay within defined limits, [control your telemetry data](/control) to manage data volume. ## License limits License limits are defined in your contract and track telemetry usage against your purchased capacity. Licensing applies across metrics, traces, logs, and change events, and is measured by using dimensions such as persisted writes, matched writes, and persisted cardinality. Organizations that purchase credits can spend a single pool of capacity across any telemetry type. Track consumption against your budgets using the [License Overview](/administer/limits-licensing/licensing), which surfaces snapshot and trend views per telemetry type. Refer to these pages for terminology about licensing in Observability Platform: * [Concepts](/administer/limits-licensing/concepts) describe licensing concepts across telemetry types. * [Definitions](/administer/limits-licensing/definitions) provide formal definitions of units like credits, persistence, and resolution. ## System limits System limits are technical constraints that protect Observability Platform from instability caused by abnormal traffic patterns. For organizations that use consumption pricing, consumption system limits restrict rapid increases in metrics, logs, and traces volume. View the thresholds, utilization, and dropped data for your tenant in the [Consumption System Limits dashboard](/observe/dashboards/managed-dashboards#consumption-system-limits). Consumption system limits apply separately from contract credit consumption and customer-configured budgets. Contact [Chronosphere Support](/support) to request an adjustment to these limits. Observability Platform also applies data validation, ingestion, and query protections for each telemetry type. See [system limits](/administer/limits-licensing/limits) for details. # Licensing concepts Source: https://docs.chronosphere.io/administer/limits-licensing/concepts Learn about telemetry data licensing concepts and usage in Chronosphere Observability Platform. Chronosphere Observability Platform supports two pricing models: consumption and capacity. Your contract with Chronosphere determines which model applies to your tenant. Each model has its own set of licensing concepts that govern how usage is measured, reported, and controlled. ## Consumption pricing In the consumption model, your organization purchases a pool of [credits](/administer/limits-licensing/licensing#credits-consumption) over a defined contract period and spends them across any telemetry type. Usage is measured in volume-based units such as persisted series, data points, and bytes, each with a per-unit credit price. The consumption model provides flexibility to shift spend between metrics, logs, and traces as operational needs change. To protect against overspending, configure [budgets](/control/consumption/budgeting) with thresholds that alert or drop data before credit exhaustion. For the full list of consumption licensing concepts and how each unit is counted, see [Consumption licensing](/administer/limits-licensing/concepts/consumption-licensing). ## Capacity pricing In the capacity model, your organization contracts for fixed rate limits measured in data points per second (DPPS) and active time series cardinality. Exceeding these limits incurs penalties that can result in dropped data. Capacity limits are enforced across three dimensions: persisted writes, matched writes, and persisted cardinality. To control which data drops first when limits are reached, configure [quotas](/control/shaping/shape-metrics/quotas) and split total system capacity into per-pool allocations. For the full list of capacity licensing concepts, metrics, and limit management strategies, see [Capacity licensing](/administer/limits-licensing/concepts/capacity-licensing). # Capacity licensing concepts Source: https://docs.chronosphere.io/administer/limits-licensing/concepts/capacity-licensing Learn about metrics licensing concepts and usage in Chronosphere Observability Platform in a capacity licensing model. In a capacity model, [quotas](/control/shaping/shape-metrics/quotas) determine which data drops first. You can split the total system-persisted writes per second into per-pool quota allocations. In a capacity model, Observability Platform defines two types of metric licenses: the **Standard Metrics License** and **Histogram Metrics License**. ## Standard Metrics License The Standard Metrics License measures aggregations, persisted writes, and persisted cardinality license consumption for the following Observability Platform metric types: * Cumulative counter * Delta counter * Gauge Because Observability Platform aggregates and persists legacy Prometheus histograms and OpenTelemetry explicit bucket layout histograms as cumulative or delta counters, these metrics consume Standard Metrics License capacity. ## Histogram Metrics License The Observability Platform histogram metric type supports both OpenTelemetry exponential histograms and Prometheus native histograms. The Histogram Metrics License measures aggregations, persisted writes, and persisted cardinality license consumption for the following Observability Platform metric types: * Cumulative exponential histogram * Delta exponential histogram Use the **Trends** tab of the [**License Overview**](/administer/limits-licensing/licensing) page to observe histogram persisted writes, matched writes, and persisted cardinality in the **Metrics consumption trends** graph. ## Capacity license metrics The following metrics apply in a capacity licensing model. ### Matched writes *Matched writes* are the number of writes per second being matched for transformation and reshaping by the Observability Platform aggregation tier. A matched write is counted for each data point matched into each aggregator rule, whether [rollup](/control/shaping/shape-metrics/rules/rollup) or [downsampling](/control/shaping#downsample-data). If a data point matches one rule, that's one matched write. If a data point matches two rules, that's two matched writes. The sum of the matched data points per second per rule equals the total matched writes. [Recording rules](/control/shaping/shape-metrics/rules/recording) aren't considered an aggregation rule for the purpose of counting matched writes. Writes also depend on your [Collector scrape interval](/ingest/metrics-traces/collector/discover/scrape-configuration#global-scrape-configuration). Increasing the scrape interval produces fewer writes, but can reduce visibility. See your current **Matched Writes** level in the [**License Overview**](/administer/limits-licensing/licensing) **Snapshot** in the **Metrics Consumption** section. On the **Trends** page, review usage over time in the **Metrics Consumption Trends** graph. #### Matched writes metrics The following metrics apply to [matched writes](#matched-writes), which are the number of writes per second being matched for transformation and reshaping by the Observability Platform aggregation tier. | Metric name | Description | | ----------------------------------------------------- | -------------------------------------------------------------------- | | `chrono_metrics_matched_writes_license_dpps_limit` | License limit for matched write DPPS by datapoint type. | | `chrono_metrics_matched_writes_license_dpps_capacity` | Capacity limit for matched write DPPS by datapoint type. | | `chrono_metrics_matched_writes_license_dpps_consumed` | Consumption rate in DPPS of matched write license by datapoint type. | Query the following metric to understand if data is actively being dropped: ```text theme={null} chrono_metrics_matched_license_dpps_dropped ``` ### Persisted writes The number of persisted writes to the Observability Platform database consists of the following: ```text theme={null} (Number of unaggregated, raw data points written to the database) + (Number of aggregated data points written to the database) ``` If you exceed 100% of your persisted writes capacity limit, data points will be dropped before being persisted. To improve and enhance performance, stability, and features, Observability Platform adds a small number of additional time series to your database. These data points aren't counted against your license. You can review your current usage in the **Persisted writes** graph on the [**License Overview**](/administer/limits-licensing/licensing) **Snapshot** page, in the **Metrics consumption** section. To see changes over time, select **Trends**, and review the **Metrics consumption trends** graph. #### Persisted writes metrics The following metrics apply to [persisted writes](#persisted-writes). | Metric name | Description | | ------------------------------------------------------- | ---------------------------------------------------------------------- | | `chrono_metrics_persisted_writes_license_dpps_limit` | License limit for persisted write DPPS by datapoint type. | | `chrono_metrics_persisted_writes_license_dpps_capacity` | Capacity limit for persisted write DPPS by datapoint type. | | `chrono_metrics_persisted_writes_license_dpps_consumed` | Consumption rate in DPPS of persisted write license by datapoint type. | For each of these metrics, you can query by `datapoint_type`, such as `histogram` or `standard`. For the `_dpps_capacity` and `_dpps_consumed` metrics, you can additionally query by `pool_name`, and `priority`. For example, the following query returns the consumption rate in DPPS of your persisted write license for `histogram` data points on the `Auth Services` pool. Because `priority` isn't specified, the query returns a series for each priority: ```text wrap theme={null} chrono_metrics_persisted_writes_license_dpps_consumed{datapoint_type="histogram", pool_name="Auth Services"} ``` If you wanted to return only high priority histogram data points for the `Auth Services` pool, specify `priority = "high"` in your query: ```text wrap theme={null} chrono_metrics_persisted_writes_license_dpps_consumed{datapoint_type="histogram", pool_name="Auth Services", priority="high"} ``` Query the following metric to understand if data is actively being dropped: ```text theme={null} chrono_metrics_persisted_license_dpps_dropped{limit="persisted_writes"} ``` ### Persisted cardinality [Matched writes](#matched-writes) and [persisted writes](#persisted-writes) measure the rate of data points per second at any given moment in time. Persisted cardinality operates differently, because it's a *cumulative* measure that calculates the count of the unique time series of the persisted writes that Observability Platform stores, seen over the last 2.5 hours. This measure is also known as Active Time Series (ATS). Persisted cardinality can be influenced by a change in ingested metrics, or you can use [rollup rules](/control/shaping/shape-metrics/rules/rollup) to downsample and aggregate metrics before they're stored. Because this measure is cumulative, reductions in persisted cardinality aren't reflected immediately, as inactive time series continue counting until they fall outside the 2.5 hour rolling window. Read more about persisted cardinality limits work and how to manage them: * [Learn how persisted cardinality limits work](#how-persisted-cardinality-limits-work) * [Manage persisted cardinality limits](#manage-persisted-cardinality-limits) * [Avoid persisted cardinality limits](#avoid-persisted-cardinality-limits) To see persisted cardinality license usage changes over time, in the [**License overview**](/administer/limits-licensing/licensing) select **Trends**, and review the **Metrics consumption trends** graph. #### Persisted cardinality metrics The following metrics apply to [persisted cardinality](/administer/limits-licensing/concepts/capacity-licensing#persisted-cardinality). | Metric name | Description | | ------------------------------------------------------- | ----------------------------------------------------------------------- | | `chrono_metrics_persisted_cardinality_license_limit` | License limit for active persisted time series cardinality. | | `chrono_metrics_persisted_cardinality_license_capacity` | Capacity limit for active persisted time series cardinality. | | `chrono_metrics_persisted_cardinality_license_consumed` | Consumption of the persisted write cardinality limit by datapoint type. | Query the following metric to understand if data is actively being dropped: ```text theme={null} chrono_metrics_persisted_license_dpps_dropped{limit="persisted_cardinality"} ``` #### How persisted cardinality limits work Persisted cardinality is comparable to a leaky bucket. Over time, new series can be added until the bucket is full. When the bucket is at maximum capacity, there's no space for new time series, so they're rejected. When existing time series expire, they make room for new series. In the following example, the persisted cardinality capacity is five unique time series. The animated image shows the lifecycle of six, unique time series (A, B, C, D, E, and F) as new data points are added, and as other data points expire. Animated image showing data points being introduced. When the persisted cardinality limit is reached, no more time series are accepted. As data points are introduced, they're either accepted or rejected based on whether the persisted cardinality bucket is full (reached maximum capacity), and whether the related time series already exists in the bucket: * If the bucket is at maximum capacity and the series already exists, the data point is accepted. * If the bucket is at maximum capacity and the series doesn't exist, the series is rejected. The following table shows how data points A3, E1, and F1 are processed, based on the bucket status: | Data point | Status | Description | | ---------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | A3 | | Time series A is in the bucket, so data point A3 is accepted. | | E1 | | Time series E isn't in the bucket, but the bucket has space for one more time series, so data point E1 is accepted in time series E. | | F1 | | Time series F isn't in the bucket, and the bucket is at capacity, so data point F1 is rejected. | Over time, data points expire based on when they entered the bucket. When data points exceed the 2.5 hour window, they're excluded from the persisted cardinality bucket. In the example, data points A1 and D1 expired, so they're excluded from the bucket. When data point C1 expires, it's also excluded. Because data point C1 is the last data point in time series C, the entire series is removed, making space for a new time series in the bucket. | Data point | Status | Description | | ---------- | ----------------------------------------- | ------------------------------------------------------------ | | A1 | | Data point A1 expired, so it's excluded from the bucket. | | D1 | | Data point A1 expired, so it's excluded from the bucket. | | C1 | | Data point C1 is expiring, so it's excluded from the bucket. | #### Manage persisted cardinality limits If your organization exceeds 100% of their Persisted Cardinality Capacity Limit, data points for any new time series not seen in the last 2.5 hours will be dropped until you're under this limit. Data points for existing time series will continue to be persisted. Series that are more stable or regularly emitted aren't at risk of being dropped because they're always in the system, and aren't categorized as new series. For example, series that don't change any labels are considered more stable. To fully resolve a penalty period, the rate of new series must be less than the rate of expiring series. The higher the differential between these rates, the faster the penalty resolves. To manage persisted cardinality limits: * Review the [Persisted Cardinality Quotas](/observe/dashboards/managed-dashboards#persisted-cardinality-quotas) dashboard, the [Usage Dashboard](/observe/dashboards/managed-dashboards#usage-dashboard) and the [Metric Growth](/observe/dashboards/managed-dashboards#metric-growth) dashboard to understand the source of cardinality growth. * Create [drop rules](/control/shaping/shape-metrics/rules/drop-rules) and [aggregation rules](/control/shaping/shape-metrics/rules) like mapping rules and rollup rules to roll away sources of growth. Old series remain in the cardinality window for 2.5 hrs. * Use the [Recommendations](/control/shaping/shape-metrics/reduce-cardinality/recommendations) page to help identify metrics and labels with no usage or utility over the past 30 days. You can then create drop rules and rollup rules based on the recommendations. The 2.5 hour expiration window is a rolling window, which means the constant rate of expiring series makes room for an equal rate of new series to be added. This behavior means the penalty period you experience can be much shorter than 2.5 hours. Queries with time ranges that overlap with penalty periods can produce inconsistent or partial results due to the dropping of time series during those periods. #### Avoid persisted cardinality limits Create [thresholds](/control/consumption/budgeting) on budgets to help manage both anomalous spikes in data and slow data growth over time. Configure actions on each threshold and set a priority to determine what data to drop, and in what order. Use the following tools and techniques to avoid hitting persisted cardinality limits: * Review the [Persisted Cardinality Quotas](/observe/dashboards/managed-dashboards#persisted-cardinality-quotas) dashboard, the [Usage Dashboard](/observe/dashboards/managed-dashboards#usage-dashboard) and the [Metric Growth](/observe/dashboards/managed-dashboards#metric-growth) dashboard to understand the source of cardinality growth. * Learn about different methods to [reduce cardinality](/control/shaping/shape-metrics/reduce-cardinality). * Proactively create [drop rules](/control/shaping/shape-metrics/rules/drop-rules) and [aggregation rules](/control/shaping/shape-metrics/rules) like mapping rules and rollup rules ahead of potential overages to evict older time series and make room for new ones. * Proactively define [thresholds](/control/consumption/budgeting) on budgets to better manage persisted cardinality. * If your organization knows which new metrics services are generating, try to control the rate that new series are introduced through smaller, more incremental deploys. ## Capacity limits Capacity limits only apply to capacity pricing. If your organization uses the consumption model, see [Manage consumption](/control/consumption). Capacity limits indicate your maximum license capacity for telemetry data in Observability Platform. Exceeding your capacity limits incurs penalties, which can result in dropped data. Dropped data can affect dashboards, alerts, and other reports. Licensing capacity is based on your telemetry types and usage. The license limit indicates your contractual license with Chronosphere. The capacity and license limits display in the **Contracts** tab of the [**License Overview**](/administer/limits-licensing/licensing) page. These limits are broken down into individual limit graphs: * **Persisted writes** * **Matched writes** * **Persisted cardinality** * **Histogram persisted writes** * **Histogram matched writes** * **Histogram persisted cardinality** Chronosphere recommends creating [alerts](/investigate/alerts) using the existing capacity limit metrics, which are also used in the [License Overview](/administer/limits-licensing/licensing). Use alerts to be notified when you're close to or over 100% of your license limit and therefore at risk of experiencing drops. ## Legacy licensing metrics The following table explains metrics that might be present in your environment, but will be replaced by new metrics. The following metrics replace this table: * [Persisted writes](#persisted-writes) * [Persisted cardinality](#persisted-cardinality) * [Matched writes](#matched-writes) These metrics create the following tags during [dashboard creation](/observe/dashboards#create-a-dashboard): * `chronosphere_service` | Metric name | [Metric type](/control/shaping/shape-metrics/types) | Description | | ----------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `limit_service_cardinality_count`
replaced by
`chrono_metrics_persisted_cardinality_license_consumed` | Counter | Current cardinality count across all Collectors. | | `limit_service_licensed_cardinality_limit`
replaced by
`chrono_metrics_persisted_cardinality_license_limit` | Counter | Current cardinality limit across all Collectors. | | `limit_service_licensed_persist_limit`
replaced by
`chrono_metrics_persisted_writes_license_dpps_limit` | Counter | Current limit for data points persisted in the database across all Collectors, as defined in the contract. | | `limit_service_capacity_limit` | Counter | Current capacity limit for data points persisted in the database across all Collectors, based on grant by Chronosphere. | | `limit_service_persisted_count`
replaced by
`chrono_metrics_persisted_writes_license_dpps_consumed` | Counter | Total number of data points persisted in database. | | `limit_service_matched_limit`
replaced by
`chrono_metrics_matched_writes_license_dpps_limit` | Counter | Current license limit for matched write DPPS by datapoint type. | | `limit_service_capacity_limit`
replaced by
`chrono_metrics_matched_writes_license_dpps_capacity` | Counter | Current capacity limit for matched write DPPS by datapoint type. | | `chronosphere_rule_metrics_matched`
replaced by
`chrono_metrics_matched_writes_license_dpps_consumed` | Counter | Consumption rate in DPPS of matched write license by datapoint type. | # Consumption licensing concepts Source: https://docs.chronosphere.io/administer/limits-licensing/concepts/consumption-licensing Learn about licensing concepts in Chronosphere Observability Platform in a consumption model. In a [consumption](/control/consumption) model, configure [budgets](/control/consumption/budgeting) to take action and protect against overspending. For each budget, define thresholds and priorities to define the actions to take when a threshold is exceeded, and control the data that gets dropped. You can then attach budgets to a [partition](/control/consumption/partitions). The consumption model includes the following concepts, which are volume-based licensing resources for telemetry data that you can spend [credits](/administer/limits-licensing/licensing#credits-consumption) on. ## Logging license consumption Spend credits flexibly across the following log resources in the consumption model: * Persisted bytes: Log bytes stored in the database. * Processed bytes: Log bytes matched for transformation and reshaping. ## Tracing license consumption Spend credits flexibly across the following trace resources in the consumption model. * Persisted bytes: Trace bytes stored in the database. * Processed bytes: Trace bytes matched for transformation and reshaping. ## Metrics license consumption Spend credits flexibly across the following metric resources in the consumption model. ### Metrics persisted datapoint A metrics *persisted datapoint* is an individual, timestamped data point that Observability Platform persists to storage. Each persisted data point counts as exactly one unit, regardless of metric type. Persisted datapoint pricing varies by effective datapoint resolution (EDR). EDR is the average time between consecutive persisted data points of a time series, computed across all persisted series and data points in your tenant. ### Metrics persisted series A metrics persisted series is a unique time series, defined by a distinct combination of metric name and labels, that Chronosphere Observability Platform persists to storage. Unlike [capacity licensing](/administer/limits-licensing/concepts/capacity-licensing#persisted-cardinality), which counts all active time series within a 2.5-hour rolling window, the consumption model counts one time for each unique persisted series when first observed. #### How persisted series counting works In the consumption model, a persisted series is counted only the first time a unique time series (a specific combination of metric name and labels) is saved to storage. The following animated image shows five instances of a service deploying at a four-minute interval. Each instance produces one unique series, for a total of five persisted series. Because each series is counted only once at first observation, the total persisted series count is five, regardless of how many data points those series produce over time. Animated image showing how persisted series are counted as five service instances deploy. Observability Platform uses a 48-hour window to determine whether a series is new. If a time series continuously receives data points, or reappears within 48 hours of its last write, it isn't counted again. Only if a series reappears after a gap of more than 48 hours is it treated as new and incurs a new persisted series charge. The following example shows an incoming series, followed by gap that exceeds the 48-hour window. Any series after that period is considered a new series, which incurs a new persisted series charge. Animated image showing how data series are calculated across a 48-hour window. In practice, the vast majority of time series receive data points continuously for their lifetime and are counted only once. The 48-hour window prevents double-counting for series with brief gaps, rather than defining what's active at any moment. #### How the consumption model differs from capacity In the capacity model, the 2.5-hour window is inclusive: all series seen within the window count toward the cardinality limit, regardless of whether they're still actively emitting. Churn inflates the count because both old and new series overlap in the window. In the consumption model, the 48-hour window is exclusive: it prevents recently seen series from being double-counted. A longer window reduces duplicate charges, and 48 hours is long enough that most series are never counted more than once. This separation means cardinality in the consumption model reflects the actual number of distinct series persisted, independent of timing effects. ### Combining persisted series and datapoints In the consumption model, persisted datapoints and persisted series form the two distinct halves of your total persistence cost, acting together to separate cardinality from volume: * Persisted series: captures how many unique time series exist. Counted once per unique series at first observation. * Persisted datapoints: captures how much data those series produce over time. Counted for every data point written to storage. The previous [animated image](#how-persisted-series-counting-works) contains five instances. With an effective datapoint resolution (EDR) of 60 seconds, if each series emits one data point per minute over a four-minute interval, the persisted data points total 20: ```text theme={null} five series * four data points per series = 20 persisted data points ``` The persisted series count remains five because each series is counted only once, while the persisted data points count grows with each data point written. ### Metrics persisted histogram datapoint bucket A metrics *persisted histogram datapoint bucket* is a bucket within a histogram data point that Observability Platform persists to storage. One persisted histogram datapoint bucket is incurred for each non-zero bucket within a persisted histogram datapoint. Only populated bucket positions in the histogram's sparse encoding are counted. ### Metrics matched datapoint A metrics *matched datapoint* is a data point matched for aggregation. One matched data point is incurred for each aggregation rule matched for each datapoint. If a data point matches one rule, that's one matched datapoint. If a data point matches two rules, that's two matched data points. [Recording rules](/control/shaping/shape-metrics/rules/recording) don't count toward matched data points. ### Metrics matched histogram datapoint bucket A metrics *matched histogram datapoint bucket* is a bucket within a histogram data point matched for aggregation. One matched histogram datapoint bucket is incurred for each non-zero bucket within a matched histogram datapoint, multiplied by the number of aggregation rules matched. # License unit definitions Source: https://docs.chronosphere.io/administer/limits-licensing/definitions Learn about licensing unit concepts in Chronosphere Observability Platform. Chronosphere Observability Platform uses the following definitions for licensing units across both capacity and consumption models: * *Control* is the volume of data that undergoes transformation and reshaping. Control is measured in `GB` for traces, `GB` for logs, and matched writes for metrics. * *Long Term Retention* is the number of months Chronosphere will retain data at a negotiated resolution. * *Persistence* is the volume of data stored following processing by control. Persistence is measured in `GB` for traces, `GB` for logs, `WPM` for events, and cardinality and resolution for metrics. * *Raw Retention* is the number of days Chronosphere will retain raw data. For more information, see [ingestion limits and retention policies](/administer/limits-licensing/licensing#contract-details). * *Resolution* is the ratio of active time series to persisted writes per second. * *Time series* is a set of data points with matching labels. ## Consumption definitions The following definitions apply only to the [consumption pricing model](/control/consumption) that uses partitions and budgets to control your telemetry data usage: * *Credits* are a flexible, fungible unit of currency that you can spend on any telemetry type without needing to renegotiate your contract. Learn more about [Credits consumption](/administer/limits-licensing/licensing#credits-consumption) to understand how credits work in the consumption pricing model. * *Effective Datapoint Resolution* (EDR) represents the average time interval between data points for your time series. * *Metrics Matched Datapoint* is a data point matched for aggregation. One Metrics Matched Datapoint is incurred for each aggregation rule matched for each data point. * *Metrics Matched Histogram Datapoint Bucket* is a bucket within a histogram data point matched for aggregation. One Metrics Matched Histogram Datapoint Bucket is incurred for each non-zero bucket within a matched histogram data point. * *Metrics Persisted Datapoint* is an individual, timestamped data point that's persisted to storage. Metrics Persisted Datapoints pricing varies by Effective Datapoint Resolution. * *Metrics Persisted Histogram Datapoint Bucket* is a bucket within a histogram data point that's persisted to storage. One Metrics Persisted Histogram Datapoint Bucket is incurred for each non-zero bucket within a persisted histogram data point. * *Metrics Persisted Series* is a time series, defined as a combination of metric name and labels, that's persisted to storage. One Metrics Persisted Series is incurred for a time series if no identical time series have been persisted in the last 48 hours. Metrics Persisted Series pricing varies by storage retention. ## Capacity definitions The following definitions apply only to the capacity pricing model that uses [quotas and pools](/control/shaping/shape-metrics/quotas) to control your telemetry data usage: * *Matched Writes* (`mW`) are the number of writes per second being matched for transformation and reshaping. For more information, see [matched writes](/administer/limits-licensing/concepts/capacity-licensing#matched-writes). * *Persisted Cardinality* is the count of unique time series of the persisted writes stored by Chronosphere. For more information, see [persisted cardinality](/administer/limits-licensing/concepts/capacity-licensing#persisted-cardinality). * *Persisted Writes Per Minute* (`WPM`) is the total count of individual, timestamped data points persisted in a one-minute interval. For more information, see [persisted writes](/administer/limits-licensing/concepts/capacity-licensing#persisted-writes). * *Persisted Writes Per Second* (`WPS`) is the total count of individual, timestamped data points persisted in a one-second interval. # Observability Platform licensing Source: https://docs.chronosphere.io/administer/limits-licensing/licensing Learn about licensing concepts and retention periods in Chronosphere Observability Platform. To track your telemetry usage against your licensing dimensions, use **License Overview**. Chronosphere provides this overview to monitor data usage against your licensing dimensions. Use the overview charts to help identify usage trends across data types and proactively manage data usage to avoid exceeding your organization's licensing limits. The **License Overview** is available in your Observability Platform tenant. In the navigation menu, click ** Go to Admin** and then select ** License Overview**. The **License Overview** page consists of two main sections, [**Consumption**](#consumption-tracking) and [**Contract**](#contract-details), selectable by tab. ## Consumption tracking The **Consumption** tab captures the current consumption rates across all license dimensions, compared to your license limits. These statistics display across two selectable tabs for **Snapshot** and overall **Trends**: * **Snapshot** is an overview of your current license usage. When selected, this page displays each licensing statistic as a percentage of your contract limit, along with a graph for that statistic over a pre-selected time period. * **Trends** explains how your license usage has changed over the selected time period, grouped and graphed by telemetry type. License dimensions currently exceeding capacity highlight in red, and licenses close to exceeding capacity are orange. ### Metrics consumption Use the **License Overview** to track trace consumption against your metrics license dimensions: * On the **Snapshot** page, the **Metrics consumption** section displays aspects of processed, persisted, and matched metrics data, measured in bytes. Panels also display this information for histogram data. These panels include: * Current percentage gauges for persisted writes, matched writes, and persisted cardinality that display consumption versus capacity. * Seven-day trend lines that plot consumption against capacity so you can spot trends from week to week. For most users, the processed and persisted consumption rates are for the current month. If your organization uses consumption-based pricing, then the processed and persisted bytes are measured against your contract to date. * On the **Trends** page, you can view metrics consumption trends in higher resolution. If your organization uses consumption-based pricing, click the link to view consumption trends in the [**Consumption**](/control/consumption/analyze) page. ### Tracing consumption Use the **License Overview** to track trace consumption against your tracing license dimensions: * On the **Snapshot** page, the **Traces consumption** section displays aspects of processed and persisted tracing data, measured in bytes. These include: * Daily average rates per second * Month-to-date cumulative trends For most users, the processed and persisted consumption rates are for the current month. If your organization uses consumption-based pricing, then the processed and persisted bytes are measured against your contract to date. To help control costs and maximize the usefulness of your tracing data, click **Go to Trace Control Plane** to [sample your traces](/control/shaping/sample-traces). * On the **Trends** page, you can view trace consumption trends in higher resolution. If your organization uses consumption-based pricing, click the link to view consumption trends in the [**Consumption**](/control/consumption/analyze) page. ### Logs consumption Use the **License Overview** page to track log consumption against your logs license dimensions: * On the **Snapshot** page, the **Logging consumption** section displays the following panels and graphs: * Percentage of persisted writes and processed writes consumed against your full licensing contract. * Elapsed time against your full contract. Compare the percentage of persisted writes and processed writes consumption against the time elapsed to understand your true consumption rate. * Daily persisted and processed log data over the past 30 days. * Cumulative trend of persisted data and processed data against the license limit, over the life of your licensing contract. To view log license information, click **Go to Consumption**, which opens the [Consumption page](/control/consumption). * On the **Trends** page, you can view log consumption trends in higher resolution. Trends display for the last seven days by default. The **Persisted** limit displays the number of logs that can persist per minute in your tenant. If your organization uses consumption-based pricing, click the link to view consumption trends in the [**Consumption**](/control/consumption/analyze) page. ### Events consumption Use the **License Overview** to track events consumption against your change events license dimensions: * On the **Snapshot** page, the **Events consumption** graph for **Persisted Capacity** displays the percentage of events consumed against your events license capacity. To filter change events by different attributes, click **Go to Changes Explorer** to open the Changes Explorer page. * On the **Trends** page, you can view event consumption trends in higher resolution. The **Persisted capacity** limit displays the number of events that can persist per minute in your tenant. This limit is enforced and can incur penalties if exceeded. In certain circumstances, this limit can exceed the license limit temporarily. The **Capacity limit** displays the number of events that can persist per minute, as defined by your the license in your contract with Chronosphere. The current consumption percentage displays a decimal ratio of persisted data against your per-minute license limit for the selected time period. This value is calculated by dividing the persisted events per minute by the number of events that can persist per minute, defined by the license limit. Use this information to understand the relationship between your persisted data consumption and the defined license limit. ### Credits consumption This feature isn't available to all Chronosphere Observability Platform users and might not be visible in your app. For information about enabling this feature in your environment, contact [Chronosphere Support](/support). To pay for telemetry data usage across the Chronosphere ecosystem, use *credits*. Credits are a flexible, fungible unit of currency that you can spend on any telemetry type. Instead of buying specific capacities, such as for metrics or logs separately, you purchase a total pool of credits over a defined contract period and choose how to spend them across any product on the platform. On the **Snapshot** page, the **Credit consumption** section includes: * **Credit consumption (full contract)**: The percentage of total contracted credits consumed to date. * **Elapsed time in contract**: The percentage of the contract period that has elapsed. Compare this against credit consumption to gauge spend rate. * **Credits cumulative trend**: A stacked line chart of cumulative credit usage over the life of the contract. * **Credit monthly consumption**: A bar chart of credits consumed per month. The **Credit consumption** section displays for any of your telemetry types that use consumption pricing and credits. For any telemetry data that uses capacity pricing, or consumption pricing without credits where you purchase bytes, this section doesn't display. If a credit contract has expired or is no longer active, a message directs you to the [Consumption page](/control/consumption/analyze) instead. To view the total number of credits available in your license for the defined contract period, click the **Contract** tab. The **Credits license** panel group includes a pricing table that lists the products enabled in your tenant, the resources included in your license, and the unit price for each resource. The pricing table lists the following resources, grouped by product: #### Metrics | Resource | Unit | Description | | ------------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Persisted series | Million time series | A time series, defined as a combination of metric name and labels, that is persisted to storage. One persisted series is incurred for a time series if no identical time series have been persisted in the last 48 hours. Pricing varies by storage retention. | | Persisted datapoints | Billion datapoints | An individual, timestamped datapoint that is persisted to storage. Pricing varies by effective datapoint resolution, which is the average time between consecutive persisted datapoints of a time series. | | Matched datapoints | Billion datapoints | A datapoint matched for aggregation. One matched datapoint is incurred for each aggregation rule matched for each datapoint. | | Persisted histogram datapoint buckets | Billion buckets | A bucket within a histogram datapoint that is persisted to storage. One persisted histogram datapoint bucket is incurred for each non-zero bucket within a persisted histogram datapoint. | | Matched histogram datapoint buckets | Billion buckets | A bucket within a histogram datapoint matched for aggregation. One matched histogram datapoint bucket is incurred for each non-zero bucket within a matched histogram datapoint. | #### Logs | Resource | Unit | Description | | --------------- | -------- | --------------------------------------------------- | | Persisted bytes | Gigabyte | Log bytes stored in the database. | | Processed bytes | Gigabyte | Log bytes matched for transformation and reshaping. | #### Traces | Resource | Unit | Description | | --------------- | -------- | ----------------------------------------------------- | | Persisted bytes | Gigabyte | Trace bytes stored in the database. | | Processed bytes | Gigabyte | Trace bytes matched for transformation and reshaping. | Because credits are fungible, your organization isn't locked into fixed capacities for individual features. Spend your credit balance interchangeably across any of the eligible resources as operational needs change. Usage can scale to accommodate short-term demand without triggering budget controls unless your organization configures them to do so. To delegate spend across teams and ensure maximum value from your credit pool, [create budgets](/control/consumption/budgeting) within consumption partitions. Budgets let each team operate within a defined allocation while preserving overall flexibility. If your organization completely exhausts all credits before the end of the defined contract period, further usage incurs additional charges per your contract terms. To ensure that your organization doesn't spend credits before the end of the contract period, use the included control capabilities to [manage consumption](/control/consumption). To analyze consumption across partitions to detect unwanted spikes and understand what's driving growth, click **Go to Consumption**, which opens the [Consumption page](/control/consumption/analyze). ## Contract details The **Contract** tab is a view of your license limits, which are defined in your contract with Chronosphere. This tab displays your licenses by limit type, with graphs for your license versus your capacity. This page also shows your ingestion limits and retention policies. *Retention policies* define the amount of time Observability Platform retains telemetry data. Contact [Chronosphere Support](/support) to configure the intervals used for your system. These policies might be based on your contract or license. *Ingestion limits* define the amount of raw data Observability Platform can ingest. View retention policies for all of your telemetry data in the **Contract** tab of the **License Overview** page. # Observability Platform system limits Source: https://docs.chronosphere.io/administer/limits-licensing/limits Learn about system limits in Chronosphere Observability Platform. Chronosphere Observability Platform applies system limits to protect itself from failures caused by abnormal traffic patterns, such as sudden, unexpected spikes in data ingestion. For organizations that use consumption pricing, consumption system limits restrict rapid increases in metrics, logs, and traces volume. The [Consumption System Limits dashboard](/observe/dashboards/managed-dashboards#consumption-system-limits) is the source of truth for the limits enforced in your tenant. Use it to view thresholds, utilization, and data dropped by system limit enforcement. Consumption system limits apply separately from contract credit consumption and user-configured budgets. Contact [Chronosphere Support](/support) to request an adjustment to these limits. Observability Platform also applies data validation, ingestion, and query protections. Learn more about the protections for the following data types: * [Events](/administer/limits-licensing/limits/event-limits): Ingest rate caps and field-level validation for change events. * [Logs](/administer/limits-licensing/limits/log-limits): Maximum individual log size and label length constraints. * [Metrics](/administer/limits-licensing/limits/metric-limits): Label size, label count, total series byte limits, and late- or future-arriving data windows. * [Queries](/administer/limits-licensing/limits/query-limits): Per-query scale protections, browser truncation limits, resource balancing, and automated source rate limits. * [Traces](/administer/limits-licensing/limits/trace-limits): Tag size, span timing validity, trace span count, and per-pod ingest caps. ## Monitor capacity limits For organizations that use capacity pricing, Chronosphere enforces [license capacity limits](/administer/limits-licensing/concepts/capacity-licensing#capacity-limits) for telemetry data. Exceeding these limits can put your organization at risk of dropping data. The type of data that's dropped depends on the limit exceeded. Exceeding your organization's license capacity limits can cause you to hit the defined system limits, which shouldn't occur under normal operating circumstances. To prevent your system from reaching system limits, use the following tools and information to ensure that your Observability Platform tenant remains under your license capacity limit at all times: * Use the [License overview](/administer/limits-licensing/licensing) to track your telemetry usage against your licensing. * Review the [cardinality dashboards](/observe/dashboards/managed-dashboards#cardinality-dashboards) to identify information you can use to help reduce cardinality. * Query [key metrics](/administer/limits-licensing/concepts/capacity-licensing#capacity-limits) to understand if Observability Platform is actively dropping data. * Use the provided tools and techniques to [avoid hitting persisted cardinality limits](/administer/limits-licensing/concepts/capacity-licensing#avoid-persisted-cardinality-limits). # Change event limits Source: https://docs.chronosphere.io/administer/limits-licensing/limits/event-limits Reference the ingest rate, request size, and field limits that apply when creating change events with the CreateEvent API. In addition to using Chronosphere Observability Platform, you can use the [CreateEvent API](/tooling/api-info/data_definition/operations/CreateEvent) to create change events. This API can ingest 300 change events per minute, per block in a single tenant. The total request size must be less than 1 MB. ## Change event fields The CreateEvent API accepts the following fields with the stated limits: | Field | Description | Required | Validation | Configurable | Failure | | ---------------- | -------------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ------- | | `title` | Brief description of the change event. | Yes | Maximum of 100 characters. | No | Error | | `source` | Identifies where the data originated. | Yes | Maximum of 50 characters. | No | Error | | `category` | Informs Observability Platform about the kind of change event. | Yes | Must be one of:
`"alerts"`
`"broadcasts"`
`"chronosphere"`
`"deploys"`
`"feature_flags"`
`"infrastructure"`
`"third_party"` | Yes | Error | | `happened_at` | Indicates when the change event occurred, which defaults to `now` if unset. | No | Must be a Unix epoch timestamp.

Can be set only to 24 hours before or after the current time. | Yes, with a restart | Error | | `type` | Differentiates change events within a particular `category` or `source`. | Yes | Maximum of 50 characters. | No | Error | | `labels` | Key values associated with a change event. | No | Maximum of 200 labels (key-value pairs) per event.

- Keys: Maximum of 500 characters.
- Values: Maximum of 5,000 characters. | No | Error | | `payload_json` | Payload associated with a change event. | No | Maximum of 100,000 characters. | No | - | | `payload_object` | Payload associated with a change event, if the original event was in key-value format. | No | Maximum of 100,000 characters. | No | - | # Log limits Source: https://docs.chronosphere.io/administer/limits-licensing/limits/log-limits Reference the license, log size, and label length limits that apply to log data in Chronosphere Observability Platform. Chronosphere logs have a variety of limits, depending on where your data is in the processing cycle. Your [license](/administer/limits-licensing/licensing) limits the amount of data you can write to and store within Chronosphere. Logs have the following size limits, which help ensure consistency and prevent issues with queries: * Individual logs exceeding 1 MB are dropped. * Label keys and values exceeding 1,024 characters are truncated to adhere to the 1,024 character limit. # Metric limits Source: https://docs.chronosphere.io/administer/limits-licensing/limits/metric-limits Learn about the limits on metrics data within Chronosphere Observability Platform. Chronosphere metrics have a variety of limits, depending on where your data is in the processing cycle. Your [license](/administer/limits-licensing/licensing) limits the amount of data you can write to and store within Chronosphere. [Metrics quotas](/control/shaping/shape-metrics/quotas) assign specific percentages of your total persisted writes limit to pools of metrics. Use quotas to limit groups of metrics and ensure Chronosphere retains the highest value metrics. See [Managing invalid metrics](/investigate/querying/metrics/troubleshooting#managing-invalid-metrics) for information about debugging invalid metrics. ## Size limits Metrics have the following size limits, which help ensure consistency and prevent issues with queries: * Label names: 512 bytes * Label values: 1,024 bytes * Max number of labels: 64 labels * Total time series bytes: 4,096 bytes If label values exceed 1,024 bytes, Observability Platform shortens the label values to accept the time series. Observability Platform shortens label values before validating the total time series byte length. When label values that exceed the maximum length, Observability Platform hashes the original value, and replaces the middle portion of the label with a hash value in this format: ```text theme={null} [Value shortened; Original=(len:1063 XXH64:aaf02cb52ed0a0fe)] ``` This strategy ensures that the original and shortened label values have the same prefix and suffix. The inserted hash value preserves uniqueness in the shortened values. The following example shows a metric with a long label value. ```text lines theme={null} # HELP application_exceptions_total Total count of application-level exceptions. # TYPE application_exceptions_total counter application_exceptions_total{exception="JDBCConnectionException",error_detail="org.hibernate.exception. JDBCConnectionException: Error calling Driver#connect at org.hibernate.exception.internal. SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:112) at org.hibernate.exception. internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate.engine.jdbc.connections.internal.BasicConnectionCreator.convertSqlException(BasicConnectionCreator. java:111) at org.hibernate.engine.jdbc.connections.internal.DriverConnectionCreator.makeConnection (DriverConnectionCreator.java:38) at org.hibernate.engine.jdbc.connections.internal.BasicConnectionCreator. createConnection(BasicConnectionCreator.java:58) at org.hibernate.engine.jdbc.connections.internal. DriverManagerConnectionProviderImpl$CheckConnectionTask.run(DriverManagerConnectionProviderImpl.java:285) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent. FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.mysql.cj.exceptions. CJCommunicationsException: Communications link failure. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect. NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl. newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl. newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance (Constructor.java:423)"} 1.0 ``` The following example shows the same metric, but with a shortened label value to allow ingestion. The hashed value indicates where Observability Platform shortened the label value. ```text lines highlight=9 theme={null} # HELP application_exceptions_total Total count of application-level exceptions. # TYPE application_exceptions_total counter application_exceptions_total{exception="JDBCConnectionException",error_detail="org.hibernate.exception. JDBCConnectionException: Error calling Driver#connect at org.hibernate.exception.internal. SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:112) at org.hibernate.exception. internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate. engine.jdbc.connections.internal.BasicConnectionCreator.convertSqlException(BasicConnectionCreator. java:111) at org.hibernate.engine.jdbc.connections.internal.DriverConnectionCreator. [Value shortened; Original=(len:1063 XXH64:aaf02cb52ed0a0fe)]makeConnection(DriverConnectionCreator.java:38) at org.hibernate.engine.jdbc.connections.internal.BasicConnectionCreator.createConnection(BasicConnectionCreator. java:58) at org.hibernate.engine.jdbc.connections.internal. DriverManagerConnectionProviderImpl$CheckConnectionTask.run(DriverManagerConnectionProviderImpl.java:285) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent. FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.mysql.cj.exceptions. CJCommunicationsException: Communications link failure. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect. NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl. newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl. newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance (Constructor.java:423)"} 1.0 ``` ## Late-arriving metrics Prometheus, OpenTelemetry, and other metric formats include one or more timestamp values for each data point. These values indicate when a sample was observed, or represent the time range of the data point. Chronosphere Observability Platform can accept late-arriving data points within a time frame, depending whether the data point matches an aggregation rule or is ingested without aggregation: * Aggregation rules accept data points from two minutes to eight minutes past the current ingestion time. * Raw data points can be written to the database up to two hours before the ingestion timestamp. ## Future-arriving metrics Observability Platform accepts incoming metrics one minute ahead of the current timestamp. If metrics arrive outside of this one-minute buffer, Observability Platform rejects those metrics and returns an error indicating that metrics arrived too far into the future. # Query limits Source: https://docs.chronosphere.io/administer/limits-licensing/limits/query-limits Learn about query limits in Observability Platform, what happens when you hit limits, and how to avoid hitting limits. Query resources are finite in any given system. Resource demand grows based on factors such as the source of, and the amount of data the query retrieves. Any very large query can request enough resources at either the database or browser level that it leaves few resources for other queries, causing them to time out waiting for resources to become available. Chronosphere Observability Platform employs several query limits at both the [browser level](#query-truncation-at-the-browser) and the [database level](#query-protections-in-the-database) to ensure a consistent user experience across the system in response to current query demand. Queries can come from one of two [query sources](#query-sources). ## Query sources In Observability Platform, query demand comes from either an *automated* or a *manual* source: *Automated sources*, such as [monitors](/investigate/alerts/monitors) and [recording rules](/control/shaping/shape-metrics/rules/recording) with regular evaluation intervals, produce a relatively predictable demand on the query resources at the database and at the browser. Queries from an automated source often request a smaller set of data than something like a dashboard query, but might incur high request volume. Although each individual query is small, the amount of query resources they're demanding at regular intervals is very large. *Manual sources*, such as loading a dashboard, running a query in Metrics Explorer, or making a direct API call, place demands on query resources at both the database and browser that are cyclical or spiky. Queries from a manual source are often exploratory, or retrieve longer time periods than automated queries. The load these queries place on the system varies significantly with the scale of data returned. Queries requesting more time series and data points require more resources to retrieve information from the database, and also can require more resources to deliver that information to the browser. Individual queries of either type demand different query resources depending on the amount of data retrieved from the database, amount of data returned the browser, or both. For example: * A query that places high load on the database but low load on the browser might request the sum of a large set of data. The query retrieves all relevant data points from the database (high database query load), but only returns a few summed data points to the browser for a system's performance monitoring workflow. * A query that places high load on both the database and browser might request the raw, unaggregated data points from all services (many unique time series) to return to the browser, such as for a debugging workflow. ### Automated source query limits Observability Platform includes the [Metrics Query Capacity Overview](/observe/dashboards/managed-dashboards#metrics-query-capacity-overview) dashboard to measure your automated query consumption against system capacity. Use this dashboard to understand how much query capacity remains as part of your budget. The dashboard includes queries from monitors, recording rules, and service accounts in the reporting metrics. Observability Platform uses [selectors per second](#selectors-per-second) and [data reads per second](#data-reads-per-second) metrics to track automated source query limits. Exceeding either of these limits results in dropping data with an HTTP `429 Too Many Requests` status code. Observability Platform continues dropping an indiscriminate subset of incoming queries to keep query traffic within defined limits. To guard against prematurely dropping queries when unforeseen spikes occur, Observability Platform doesn't start dropping data until either of the query limits are exceeded for 10 minutes consecutively. #### Selectors per second These metrics track the count of selectors that queries issue per second. For example, this metric includes a single selector named `up`: ```text theme={null} up{app="webserver"} ``` A more complex query might include multiple selectors. The following query includes two `sum` selectors: ```text wrap theme={null} sum(rate(http_server_handled_total{status="200"}[2m]) / sum(rate(http_server_handled_total[2m])) ``` The Metrics Query Capacity dashboard displays the number of selectors consumed and dropped against the limit per second. For more information about selectors, see the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). The selectors per second query limit can't be increased. ##### Reduce selectors per second query load Use the following strategies to reduce query load for selectors per second: * Configure longer intervals for [monitors](/investigate/alerts/monitors) and [recording rules](/control/shaping/shape-metrics/rules/recording). For example, consider 1,000 monitor queries that include a single selector, each of which runs every 15 seconds. These figures calculate to roughly 66 selectors per second (`1000/15`). Increasing the execution interval to 60 seconds reduces the number of selectors per second to roughly 16 (`1000/60`). * Structure alert and monitor queries to use PromQL aggregations for optimal efficiency. For example, you might create two separate alerts for service A and service B error rates: ```text theme={null} sum(rate(http_server_handled_total{service="A". status=~"5.*"}[1m])) > 10 sum(rate(http_server_handled_total{service="B". status=~"5.*"}[1m])) > 10 ``` Instead, create a single monitor that checks both services: ```text wrap theme={null} sum(rate(http_server_handled_total{service=~"A|B". status=~"5.*"}[1m])) by (service) > 10 ``` #### Data reads per second These metrics track the amount of raw data that Observability Platform fetched to run the specified queries per second. Observability Platform calculates data consumed by a query with this calculation: ```text /DATAPOINTS_RETURNED_FOR_SERIES/ theme={null} READS = sum(DATAPOINTS_RETURNED_FOR_SERIES) ``` * `DATAPOINTS_RETURNED_FOR_SERIES` is the number of data points retrieved for query execution. This equation calculates the sum of all data points for a series, with a minimum of 60 data points per series. All smaller values are rounded up to `60`. ##### Examples Consider a range query `sum(up{app="webserver"})` with a time range of `[now()-1h, now()]`. If the query selects 10 steady series with a scrape interval of 30 seconds, the total number of retrieved data points is 1,200, as defined by this equation: ```text theme={null} READS = 10 * (1h/30s) ``` Next, consider an instant query `up{app="database"}` that selects 20 time series with a scrape interval of one minute. Prometheus fetches five minutes of data by default, so the resulting number of fetched data points is `100`, as defined by the following equation. However, this equation doesn't apply the minimum number of data points retrieved per series, which is 60, so this equation isn't correct. ```text theme={null} 20 * 5 = 100 ``` After applying the minimum number of data points retrieved per series, this rule actually consumes 1,200 reads, as defined by this equation: ```text theme={null} 20 time series * 60 data points retrieved = 1200 reads ``` The following instant query is more advanced. This query requires Observability Platform to fetch all data points for the selected series in a 30-day period, and apply the minimum number (`60`) of data points retrieved per series. ```text theme={null} sum_over_time(job_execution_errors[30d]) ``` Queries that fetch [downsampled data](/control/shaping/shape-metrics/downsampling) will consume less data reads because datapoint resolution for the selected series is reduced, meaning less data points are used to process the query. ##### Reduce data reads per second Use the following strategies to reduce the consumption of data reads per second: * Write queries with a shorter time range or shorter range selectors. * Create [rollup rules](/control/shaping/shape-metrics/rules/rollup) for unnecessary labels in your most expensive queries to reduce the amount of data points the query needs to fetch. * Ensure that query selectors are as precise as possible, especially in join queries. For example, consider this query: ```text wrap theme={null} sum(rate(container_cpu_seconds{cluster="prod"}[1m]) by (node, container) / on (node) node_cpu_capacity ``` Including `{cluster="prod"}` in the latter part of the query reduces the number of series and in turn data points the query needs to fetch for the right side of the join. Use this instead: ```text wrap theme={null} sum(rate(container_cpu_seconds{cluster="prod"}[1m]) by (node, container) / on (node) node_cpu_capacity{cluster="prod"} ``` ## Optimize queries Large queries might be good candidates for optimization by decreasing the amount of data the query is trying to retrieve. Use the [Metrics Query Capacity Overview](/observe/dashboards/managed-dashboards#metrics-query-capacity-overview) to identify large queries. Optimize queries by: * Shortening the time window, or using [rollup rules](/control/shaping/shape-metrics/rules/rollup) to decrease the data scale. * Improving the query syntax by removing regular expressions to reduce the index lookups performed by the query. * Adjusting the number of concurrent unique requests in the system. ## Log query rate limits When [querying log data](/investigate/querying/query-logs), be mindful of the following limits that can impact your search experience, what happens if your tenant exceeds limits, and how you can avoid hitting these limits. * **Global query limit**: Each Observability Platform tenant has a global limit on the total compute resources available for queries. If this limit is reached, queries are queued and will run as soon as resources become available. * **User query limit**: This limit prevents a single person from consuming all available query resources. The user query limit is calculated based on the compute resources required to run individual queries in addition to the number of user queries that are run. Each user is limited to a percentage of the total available query resources. * **API query limit**: API queries are limited to a percentage of the total available query resources. ### Avoid log query rate limits If you experience slow or lagging queries on your log data, implement the following changes to help increase the speed of your queries: * Close unneeded tabs in your browser. Open tabs with long-running queries or that refresh often, such as in Logs Explorer or in dashboards, consume more resources. * Write efficient queries. Ensure your log queries are efficient so that they complete faster and make resources available for other queries to run. * Include the primary key in your query: Query on one of the predefined [keys](/investigate/querying/query-logs/query-syntax#keys), such as `service` or `severity`, and then use the [query syntax](/investigate/querying/query-logs/query-syntax) to incorporate additional attributes. * Narrow the query scope. Reduce the time range or scope of your query to return results faster. ## Query truncation at the browser Observability Platform can truncate queries that return many unique time series or data points to the browser to reduce the chances of long load times and query timeouts. Truncating queries protects the browser from crashing when querying many metrics. Dashboards and the Metrics Explorer both use query truncation. Observability Platform calculates the browser limit using a combination of time series, time granularity, and time period (and their resulting data points) requested by the query. The limits are: * Time series a query can return to the browser: 2,500 * Data points a query can return to the browser: 300,000 The number of time series and data points returned to the browser might be the same as the number requested from the database. If the query is doing some level of aggregation, the number of time series and data points returned might be fewer than what the query requests, but not greater. To reduce the number of series returned to the browser, view the [Aggregation Rules UI](/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules) to determine if a queryable aggregate metric already exists. If an aggregate metric doesn't exist, use [aggregation rules](/control/shaping/shape-metrics/rules) or [derived metrics](/investigate/querying/metrics/derived-telemetry/derived-metrics) to: * Query an aggregated subset of the raw data. * Break the query into smaller chunks (by time, for instance) to reduce returned data points. ### Enable or disable query truncation By default, Observability Platform enables query truncation. To disable or re-enable query truncation: 1. In the displayed dialog, select **Options**. 2. Click the **Truncate expensive query results** toggle. 3. Click **Done**. Disabling query truncation causes expensive queries to take longer to load and can result in timeouts (no results returned for a single query), or might cause the browser to crash. The Options dialog with the Truncate expensive query results toggle. ## Query protections in the database Observability Platform employs protections against very large queries in the database layer, in addition to the browser. A query that requests many unique time series or data points from the database can experience these situations: * Per-query scale protections * Across-query resource balancing protections * Individual query timeouts These protections ensure a positive user experience for the maximum number of users. A query requests resources from the database until it has the data it needs, or until resources run out. Available resources can run out due to these reasons: * Individual query scale protections are in place. * There are many queries sharing scarce query resources (query balancing). * An individual query reached a timeout. ### Query scale protections Query scale protections ensure a positive user experience for the maximum number of users. Current database query scale protections are: * [Time series](https://prometheus.io/docs/concepts/data_model/#notation) a query can retrieve: 300,000 * Standard data points a query can retrieve: 200,000,000 For Chronosphere [histograms](/investigate/querying/promql/apply-functions#querying-histogram-metric-types), the query limit depends on the number of buckets containing data within a histogram. The data points needed to satisfy the query's time requirement are calculated as: ```text wrap theme={null} (Number of requested time series) * (Data resolution of the requested time series) ``` For example, a query does a sum of data points over two minutes, storing the requested data with a 10-second resolution. That sum operation requires 12 data points per time series, calculated as: ```text theme={null} 60s + 60s = 120s; 120s/10s resolution = 12 data points ``` If the query requests 20,000 time series, the number of data points is 240,000, calculated as: ```text wrap theme={null} 20,000 time series * 12 data points per series for the 2 minute window ``` Dashboard queries request more data points because they use longer ranges. If the previous example query is a dashboard query and the dashboard window is set to the past one hour, the data points retrieved are calculated as: ```text theme={null} 60 minutes / 2 minute sum interval = 30 intervals 30 * 240,000 = 7,200,000 data points ``` Individual queries requesting more than the allowable limits will time out. Exceeding query scale limits can produce one of the following error messages: ```text theme={null} query processing would load too many samples into memory in decoding ``` ```text theme={null} The query exceeded the allowable resource limit. ``` These errors indicate that the query is trying to retrieve too many resources. Modify the impacted query to reduce the scope and reduce the number of results being fetched. ### Query resource balancing Observability Platform tracks the number of concurrent queries received, the scale of each query, and the unique users for each query. Observability Platform makes an effort to fairly balance the available query resources between large and small queries, and also between unique users. If there are many concurrent queries, and each with relatively high scale, some queries might need to complete in stages. This means that some queries must wait (be throttled) to allow queries from other unique users to make use of the available resources. The more unique users, the more balanced the system is in enabling different queries to complete. ### Query timeouts Even with balancing measures in place, if there are insufficient resources for a query to complete within a specified time period, the query will time out. Observability Platform displays an error message indicating the query timed out. If a query times out, try these methods to resolve the issue: * Modify the query syntax to retrieve less data per query. For example, try to shorten the time range. * Use shaping rules such as [aggregation rules](/control/shaping/shape-metrics/rules) to downsample or roll up unnecessary labels, and then modify the query to retrieve aggregated data points instead of the raw data points. # Tracing ingest limits Source: https://docs.chronosphere.io/administer/limits-licensing/limits/trace-limits Reference the tag, span, and trace ingest limits for tracing data in Chronosphere Observability Platform. Chronosphere uses the following ingest limits for tracing data. If you exceed one or more of these limits, Chronosphere truncates or rejects the data depending on the limit itself. Exceeding a limit indicates that you might need to modify your client-side instrumentation, or implement head or tail [sampling rules](/control/shaping/sample-traces) to drop data that you don't want to persist. The following limits are in increasing order of granularity from most granular to most broad. ## Invalid tags Chronosphere limits the size of a tag to 200 bytes for any span. Chronosphere accepts and processes the first 200 bytes and truncates any additional bytes. The related span and trace aren't otherwise impacted by this limit. ## Invalid spans If the start time of a span is greater than 10 minutes before or after the current time, Chronosphere marks the span as invalid and drops it. The parent trace isn't impacted, provided that the other spans in the trace have valid start times. ## Invalid traces Chronosphere persists individual traces with 100,000 spans or fewer. If a trace has more than 100,000 spans, Chronosphere rejects the trace and all included spans. Although Chronosphere rejects invalid tracing data, you can view rejected data in the Trace Control Plane, or by querying the `chrono_trace_dropped_volume_in_bytes_count` metric. You can also [create a monitor](/investigate/alerts/monitors#create-a-monitor) for this metric and generate notifications when tracing data is rejected. The following query returns invalid trace data by reason, such as when a span is too far in the past: ```shell theme={null} sum(rate(chrono_trace_dropped_volume_in_bytes_count[5m])) by (reason) > 0 ``` ## Pod limits Chronosphere limits data ingest to 2 GB per minute for each Kubernetes pod that handles data ingest. Chronosphere scales these pods automatically to handle current traffic, and imposes this limit to protect against sudden traffic surges to a single pod. Your tenant might have anywhere from two to 100 pods at any given time, depending on current load. The total ingest limit in one minute depends on the number of pods in use times the 2 GB per minute for each pod. ## System limits Chronosphere scales as rapidly as possible to manage sudden changes in trace data volume. However, sudden spikes might trigger surge protections that drop trace data until more resources are available. This system limit allows the majority of normally observed data fluctuation patterns and guarantees the reliability of the Chronosphere trace ingestion pipeline in the case of unexpected volume changes. Trace data volume exceeding this limit is visible in the Trace Control Plane as a metric tracking volume of traces dropped due to limiting. # Configure pinned scopes Source: https://docs.chronosphere.io/administer/pinned-scopes Learn how to configure pinned scopes in Observability Platform. [Pinned scopes](/navigate/pinned-scopes) define a set of consistent criteria for users to select a subset of data. Scopes are used in several sections in Chronosphere Observability Platform. Click the **Pin scopes** box in a supported page to display the configured scopes. To select a scope, click it. ## Create a pinned scope To create a scope: 1. In the navigation menu, click ** Go to Admin** and then select **Platform > Scope Configuration**. 2. Click ** Create**. 3. Add values to the following fields: * **Scope category name**: A name by which to group your filters. For example, the category `Environment` might contain filters for `dev`, `test`, and `production`. * **Label name**: A label from a metric, which you want to filter by. * **Matcher**: A list of label names. Matchers support regular expressions. * **Default value**: Set an optional default value for this pinned scope. This value should be one of the values from the **Label name**. * **Values**: Label values. In the previous example, `dev`, `test`, and `production` are label values. 4. Click **Save**. ## Edit a scope To edit a scope: 1. Click the three vertical dots icon in the row with the scope name. 2. Click **Edit**. 3. Update the scope values. 4. Click **Save**. ## Delete a scope To delete a scope, click the three vertical dots icon in the row with the scope name, and then click **Delete**. # Manage service configuration Source: https://docs.chronosphere.io/administer/service-discovery Discover and configure services in Chronosphere Observability Platform. This feature isn't available to all Chronosphere Observability Platform users and might not be visible in your app. For information about enabling this feature in your environment, contact [Chronosphere Support](/support). The **Services** page provides a view into your applications based on [services](/observe/services). If a service doesn't match the rules that you define, you can edit the service configuration in Observability Platform to include or remove services from your display. You can also modify services' visual presentations of your data. For setup instructions, see [Configure services](/observe/services/configure-services). ## Understand service configuration The **Service Configuration** page lets you configure how to discover services that exist in your environment, and what visualizations to present on your services pages, based on their telemetry data. All users can view service configurations and preview changes. Only users with [SysAdmin permissions](/administer/accounts-teams/teams#add-a-role-to-a-team) can save new or edit existing service configurations. ```mermaid actions={false} theme={null} graph LR PM[Service pod mapping] --> DJ["Discovery jobs"] DJ --> MR["Merge rules"] MR --> Prev["Preview"] Prev --> Present["Presentations"] ``` ## View service configuration In the navigation menu, click ** Go to Admin** and then select **Platform > Service Configuration** to open the **Service Configuration** page. ### Review changes If there are unsaved changes, the **Review & apply changes** button indicates the number of unsaved changes. An alert icon indicates modified jobs that haven't yet been saved. ### Global mappings Global mappings let services be discovered by jobs using metrics that have a Kubernetes Pod or instance label but no obvious service labels. Discovery jobs can be configured to join their [Pod](https://kubernetes.io/docs/concepts/workloads/pods/) label with the pod label from one of the global mapping configurations that has a label that's more likely to map to a service name. For example, the `kube_pod_info` metric from [kube-state-metrics (KSM)](https://github.com/kubernetes/kube-state-metrics) have a `created_by_name` which is often a service name. Chronosphere uses those labels to match the physical pods to a discovered service. Users can't edit configurations in the **Global mapping** tab. These configurations are commonly from standard KSM and OpenTelemetry libraries. Introducing new potentially conflicting configurations here could cause services to be silently ignored by discovery jobs. * **Slug**: The slug name of the discovery rule defining the service pod mapping. * **Name**: The rule name. * **Service labels**: One or more labels that could be a candidate for the service name. * **Pod label**: The pod label is usually the name of the pod or instance and is used as a key to map to the service name. These are created during [metrics discovery jobs](#use-discovery-jobs-to-identify-and-add-services). Click the three vertical dots icon and then **Open in Metrics Explorer** to open the query for that service instance in [Metrics Explorer](/investigate/querying/metrics/explorer). ### Configuration status The **Configuration Status** tab title includes the number of changes since the last run of a discovery job. If there are problems with discovery jobs, the tab includes an orange dot in the corner. Click the tab to review all messages. Use the search box to find specific messages. Each message contains the following details: * **Severity**: The severity of the alert. * **Type**: The type of process causing the error. For example, type of `discovery job` means a [discovery job](#use-discovery-jobs-to-identify-and-add-services) has errors or conflicts that need correction. * **Name**: The name of the item in the **Type** field. * **Message**: Details about the error or conflict. Use these details to review your configuration and make changes. * **Labels**: A list of labels and values that explain the conflicting rule: * `service_name`: The service with the conflict. * `first_service_name`: The first instance of a service with the label causing the conflict. * `rule_slug`: The immutable ID of the rule for this discovery job. * `source`: The label whose value causes the conflict. * `dimension_value`: The value of the conflicting label. ## Use discovery jobs to identify and add services *Discovery jobs* are templates that match metrics to a discovered service. Chronosphere provides some common discovery jobs. Create your own jobs or copy and customize existing jobs for your environment. Use the **Search jobs** box to find a specific template. * **Status**: Toggle the switch to enable or disable the job. * **Managed by**: Jobs are manged by Chronosphere (non-editable), or by the user (editable). * **Job name**: The user-readable discovery job name. * **Type**: Users can create jobs to discover services based on metrics (`Metrics`) or change events (`Events`) types. `Traces` are managed by Chronosphere and not user creatable or editable. `Traces` is used by the **Trace Services Based on Service Attribute** job, and is available only to customers with a Tracing license. Users can enable or disable the discovery job. * **Service label mapping**: Labels identifying the service name. Provided by the the service label or instance label + identifying label (**Service template label**) from the template form. * **Services discovered**: The estimated maximum number of discovered services. This number can be significantly different (often higher) than the actual number of discovered services, depending on [merge rules](#use-merge-rules-to-correct-or-remove-services). ### Create a discovery job To create a discovery job: 1. On the **Service Configuration** page, click **Create**. 2. From the menu, choose **Metrics discovery** or **Change events discovery**. 3. Add a **Job name**. 4. Select **Manual** or **From dashboard JSON**. 5. Add **Job details**: For a **Metrics discovery job**, add: * **Metric name**: Observability Platform queries this metric to find labels which contain candidate service names based on the discovery job configuration. * **Metric name prefix**: A prefix of metric names to used filter and scope metrics this job applies to. This field uses [glob syntax](/investigate/querying/glob-syntax) to match metric names. When a service is discovered by this discovery job, metrics with this prefix will receive a [derived label](/investigate/querying/metrics/derived-telemetry/derived-labels) called `lens_service` which is set to the discovered service name. This prefix, along with optional target label names in this job, must not conflict with the prefix in another enabled discovery job. * Optional: Click **Options** to add one or more **Target names** and **Target values**. These fields are respectively a label name and value [glob expression](/investigate/querying/glob-syntax) that further scope to which metrics Observability Platform applies the `lens_service` label. Use these fields to help scope the discovery job and avoid conflicts when you need to configure different discoveries on metrics with the same prefix. * **Service mapping**: Focus your template using one of the following groups: * **Service label**: A label containing the service name. This label must be found on the previously specified **Metric name**. This is the preferred option when you already have a good service name label on your metrics. Click the Click the icon to draft a `cardinality_estimate by () ()` query in [Metrics Explorer](/investigate/querying/metrics/explorer). * **Pod mapping**: Pod mappings are an advanced option. Use if you don't have an obvious label to use as the service name, but you have a pod or instance label that could join with a global mapping that has a service name. * **Instance label**: The label containing the instance or pod name. Label values are used as keys to map to the service name. For example, GRPC metrics have an **instance label** of `instance` and an identifying label of `grpc_service`. * **Transform regex**: A [regular expression](/investigate/querying/regular-expressions) that modifies the value in the instance label to match a value in the service instance mappings. This is often used to remove a pod name's prefix. Can be `(+.)` if no trimming is required. * **Transform replacement**: The regular expression replacing the original label, allowing modification of the value in the instance label so that it matches a value in the service instance mappings. You can use symbols like `$1` to insert regular expression capture groups. * **Identifying label**: A label containing information that uniquely identifies which service the time series belongs to. The identifying label can't be the `instance label` or other ephemeral label. It must be static, finite, and uniquely identify the service. For example, an identifying label value of `X` that indicates it belongs to `service1` must always map to `service1`. The value `X` can't map to another service. This mode of service discovery fails if this label is [high cardinality](/control/shaping/shape-metrics/reduce-cardinality). For an **Events discovery job**, add a **Service label**, which is the label containing a the service name. Discovery jobs use this to find service name candidates. Use the **Preview events** graph to select a service and preview the events to show. For example, if you're sending an event to Observability Platform with the payload `"pipelineName": "PIPELINE_NAME"` as a label, you can search through the service labels to find and select that label. If the event doesn't have the `lens_service` label, you can create a [merge rule](#use-merge-rules-to-correct-or-remove-services) to map the existing label to the service. Events corresponding to that label display in charts on the applicable service's page. You can create both a presentation and a metrics discovery job from a dashboard. 1. Click **Import JSON file** to import an Observability Platform standard dashboard from its [JSON representation](/tooling/api-info/dashboard_schema). This feature isn't compatible with classic dashboards. 2. Optional: Click **Select dashboard** to automatically generate the JSON representation for an existing dashboard. If the JSON is valid, Observability Platform confirms this. 3. If the dashboard is confirmed to be valid, click the **Choose service variable** dropdown that appears and select an appropriate variable. To complete discovery job completion: 1. Click **Create**. 2. Click **Review & apply changes**. ### Edit or delete a discovery job You can edit or disable existing user-created jobs to ensure your configurations match your services when those services change. 1. Click the three vertical dots and then select **Edit config** to change or **Delete** to remove the job. 2. Available edits to job details include: * Clear the **Enabled** checkbox to disable the job. * Update the **Service label**. * For metrics discovery, update the **Pod mapping** fields. 3. Click **Stage changes** to preview your updated configuration. 4. Review your changes. 5. Click **Review & apply changes** to save. When editing a metrics discovery job, you can review the **Service discovery query**, which is the query used to generate the discovery job. Click the icon to open the query in [Metrics Explorer](/investigate/querying/metrics/explorer). ## Use merge rules to correct or remove services When a service doesn't match expectations, create *merge rules* to correct mismatches. Merge rules define how to group or match service data when service names or labels might not be the same across multiple instances. Merge rules affect the number of displayed services in service pages. For example, Trace Explorer might find a particular set of services, but a service page might have a lower number of services based on the merge rules applied to those services. Use the **Search rules** box to find a specific rule, or filter by **Rule type**. The **Merge rules** table displays information that was set during rule creation. Merge rules apply from the top to the bottom of the list, in **Order**. A service matches the first rule that applies to it, and won't match additional rules. Use the drag icon to move the rule up or down in the list to change the order that rules apply. Toggle the **Status** icon to disable a job. ### Preview and apply changes This table is a list of changes that will take effect when you apply any staged merge rule updates. To display only the staged changes, click the **Show diff only** toggle. If **Review & apply updates** displays a **0** in the button, no user-created changes are staged to apply. However, the preview pane can still display staged changes. These changes are due to discovery jobs run by Observability Platform that haven't yet been applied. #### Add or modify services To run discovery jobs: 1. Click **Review & apply changes** to open a larger **Review & apply changes** drawer. This drawer lets you review a more detailed preview, including any **Presentations** attached to services. Click the **Use freshest data** toggle to run the preview without a cache from previous discovery runs. 2. Click **Run service discovery**. You must have [SysAdmin](/administer/accounts-teams/teams#add-a-role-to-a-team) permissions to run service discovery. In the **Review & apply changes** drawer, the number of unchanged, added, deactivated, and modified services displays under the drawer title. Select **Skip service discovery** to wait until the next schedules discovery run. To discard changes, click **Reset changes**. ### Create a merge rule You can merge a limited number of service names together. Merge rules can be used to group service names like `gateway` and `gateway-staging` together. They can't merge high cardinality service names together like `gateway-1` and `gateway-2` to `gateway-n`. It's always better to have a service label that already uses the correct service name rather than using merge rules to fix them afterwards. Merge rules use [regular expressions](/investigate/querying/regular-expressions). To create a merge rule: 1. In the **Merge rules** tab of **Service Configuration**, click **Create** and then select a rule type: * **Ignore** rules remove matching services from the interface, stop data collection for them, and prevent discovery from adding new services that match this rule. * **Merge** rules group services together. * **Suppress**: Use this to prevent noisy service discovery jobs from creating too many ephemeral services in your environment. Adding a **Suppress** rule for a given **Suppressed Discovery Slug** means that this discovery job won't discover new services. If a service is discovered by another discovery job, presentations configured to display for the suppressed discovery job will display. 2. Add the following fields: * **Rule Description**: Describe what this service rule does. Displays on the configuration page as **Description**. 3. The following fields display based on your rule type: * **General Settings**: The **Rule description**. * **Rule details**: The following fields display in the **Details** section of the **Merge rules** page. * **Source labels**: Specify which label's values the rules apply to. The `Discovered service name` is a source label, and is most commonly selected. * **Regex**: Add a new regular expression to replace the original. On the service configuration page, the original **Regex** displays, along with the **Replacement**, which is the value added here. * **Suppressed Discovery Slug**: When using a **Suppress** rule, add a discovery slug to suppress new services from being generated by that discovery. * **Global Suppression Exceptions**: When using a **Suppress** rule, don't suppress the listed services, even if they're generated by suppressed discoveries. 4. Click **Stage changes**. The **Service configuration** page displays, with your updates displayed. 5. The **Preview** updates. Preview your changes using the table. The **Service name** and **Discovery jobs** included in that service show what metrics are merged into a given service name. Services and metrics being removed from display have a **`[-]`** before them. 6. Click **Review & apply changes**. #### Remove a service or prevent service creation You can deactivate a service by removing the discovery rule that identifies it. Observability Platform doesn't display an inactive service on the **Services** page and no data is collected for it. To deactivate specific services, or to prevent new services from being created that match certain criteria, create a narrowly defined [ignore merge rule](#create-a-merge-rule) that matches the unwanted service. For example, you can use ignore merge rules to remove services that lack data from view. Such services persist as inactive services. You can force the **Services** page to display them by enabling the page's **Include inactive services** toggle. The page of an inactive service also displays a banner indicating its inactive state. ### Edit or delete a rule To edit or delete a service rule: 1. Click the three vertical dots icon. 2. Click **Edit rule** to change the rule, or **Delete** to remove the rule. 3. Edit the fields, per their descriptions in the [create rules](#create-a-merge-rule) section. 4. Click **Stage changes**. The **Service discovery preview** displays your updates. 5. Click **Apply changes** to save or **Reset changes** to cancel. ## Presentations Presentations are templates for how service data displays. Observability Platform provides default presentations for your use. You can also create your own custom presentations. ### View presentations To review existing presentations, on the **Service Configuration** page, click **Presentations**. All of your presentations display as cards separated into the following groups: * **Chronosphere Managed**: Default presentations, which can't be edited. * **Custom Presentations**: Editable presentations created by users. Each card displays the following information: * **Name**: The presentation name. * Enabled status: Either `Enabled` or `Disabled`. * **Number of Services**: The number of services this presentation represents. * **Number of panels**: The number of dashboard panels this presentation displays. ### Create a presentation You can create custom presentations for your services. Presentations can use any panel type that a [Chronosphere dashboard](/observe/dashboards/panels) can use. To create a presentation: 1. In the **Presentations** page, click **Create**. 2. Add a **Name** and **Description** to identify the presentation. 3. Create conditions the presentation must meet. In the **IF service has** menu, select a discovery job. The presentation will show on services that have been discovered by this discovery job. 4. Click **+ Add condition** to **show** more services, or to **hide** specific services from this presentation. 5. Review the list of services that match the selected conditions. Make changes to the conditions if needed. 6. The **Review Content** section of the creation page defines how the content displays on the [service page](/observe/services/service-pages). 7. Click **Start with existing dashboard** if you already have a dashboard created to use. Click **Start a new view** to create a new dashboard, which opens the dashboard editor in a new tab. #### Use an existing dashboard If you have a dashboard for an existing service, and you're creating a service presentation, you can select the service as a variable. Observability Platform replaces that variable in the JSON with `lens_service` to make the presentation flexible for all services, which lets you reuse this presentation for multiple services. To import an existing dashboard: 1. Click **Start with existing dashboard**. 2. Select a [dashboard](/observe/dashboards) from the list. 3. Click **Save** when finished. 4. Click **Review & apply changes**. #### Start a new view If you select **Start a new view**, the dashboard editor opens in a new tab. 1. Add [panels](/observe/dashboards/panels) and [variables](/observe/dashboards#create-and-configure-dashboard-variables) to your new dashboard. 2. Click **Save** when finished. 3. Click **Save** in the **Create** screen. 4. Click **Review & apply changes**. ### Edit a presentation To change an existing presentation: 1. On the **Service Configuration** page, click the three vertical dots and then select **Edit presentation**, or on the **Presentations** page, click the card for the service. 2. Make changes to your presentation. You can modify the presentation's **Name** and **Description**, amd **Edit Conditions**. You can also **Review services which match the conditions**. 3. Click **Save**. ### Delete a presentation To delete a presentation: 1. [Edit the presentation](#edit-a-presentation). 2. Click **Delete presentation**. 3. Click **Delete** to confirm that you want to delete the presentation. ## Create SLOs from discovery jobs After discovery finds and displays services, you can create [service level objectives](/observe/slo) (SLOs) tied to a metrics discovery job from **Service Configuration**. ### Edit SLO config To create availability or latency SLOs for a metrics discovery job: 1. On the **Service Configuration** page, on the **Discovery jobs** tab, click **See more actions** () on a metrics discovery job row, and then select **Edit SLO config**. 2. Configure the SLO using the same fields described in [Define an SLO](/investigate/alerts/manage-slos#define-an-slo). 3. Click **Save**. **Edit SLO config** appears only for metrics discovery jobs, not event discovery jobs. For the queryless RPC and gRPC builder and other creation paths, see [Create a new SLO](/investigate/alerts/manage-slos#create-a-new-slo). # Chronosphere compliance information Source: https://docs.chronosphere.io/compliance Information about Chronosphere compliance standards Chronosphere is SOC 2 Type 2 and ISO 27001 audited. To request our report, reach out to your assigned account manager, or contact [Chronosphere Support](/support). For more information about Chronosphere security, see the [Security overview](https://chronosphere.io/security-overview/). # Control your telemetry data Source: https://docs.chronosphere.io/control Learn about how to control your telemetry data with Chronosphere Observability Platform. Your app generates a lot of data, but not all of it's actionable. Storing every byte is costly, and can quickly overwhelm your capacity limit, license limit, or both. Aside from financial cost, there's computing cost associated with generating, processing, and temporarily storing your telemetry data. Reducing telemetry data volume keeps costs predictable and manageable as your business scales, and helps teams manage their data effectively by reducing the amount of available information. When you optimize your telemetry data, on-call engineers can find the data they need to solve problems faster. To take control of your telemetry data, Chronosphere Observability Platform provides multiple tools you can use to identify and eliminate data that isn't used and doesn't deliver value to your business. See [Data control concepts](/control/storage) to learn about the mechanisms you can use to control your telemetry data. The following diagram shows how the stages of the control workflow connect, and each stage links to its section on this page. The workflow is a loop rather than a one-time setup, because partitions and budgets reveal what your data consumes, which sends you back to your shaping rules. For a guide that defines this workflow, including how to set thresholds that alert or drop data, see [Manage telemetry data budgets](/overview/guides/manage-budgets). ```mermaid actions={false} theme={null} flowchart LR Send["Send data"] subgraph OP["Observability Platform"] Reduce["Reduce"] Shape["Shape"] Review["Review"] Partition["Partition"] Analyze["Analyze"] Budget["Budget"] end Send --> Reduce --> Shape --> Review --> Partition --> Analyze --> Budget --> Shape click Reduce "#reduce-volume" click Shape "#shape-and-sample-your-data" click Review "#review-rules" click Partition "#create-partitions" click Budget "#create-budgets" click Analyze "#analyze-consumption" classDef rules fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF; class Reduce,Shape,Review,Partition,Budget,Analyze rules ``` ## Reduce volume After sending data to Observability Platform, use the [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer) to identify opportunities to reduce the overall volume of telemetry data. If you're sending Prometheus metrics, you can also use the [](/investigate/analyze/usage) to identify obsolete or unnecessary data, and understand the impact of a proposed shaping rule to users of that data. ## Shape and sample your data Use the Chronosphere Control Plane to [shape and sample your telemetry data](/control/shaping) and reduce the amount of data you persist. After implementing partitions and budgets to manage consumption, you can revisit control rules to transform, reshape, retain, or exclude telemetry data. ## Review rules After creating shaping rules, use the various tools that Observability Platform provides to [review the impact](/control/shaping/review-impact). If your shaping rules reduced too much data or not enough, make changes to ensure the rules achieve the intended outcome. ## Create partitions To help [manage telemetry data consumption](/control/consumption), create *partitions*, which are slices of your data with distinct owners. Partitions provide a consistent structure for attributing usage and costs to the appropriate owners in your organization so they can isolate and control independent their parts of the business. ## Analyze consumption After defining partitions, use the [Consumption page](/control/consumption/analyze) to view persisted log data consumption against license capacity. This page helps to detect unwanted spikes and understand what's driving growth across your telemetry data. Use this information to [review shaping rules](#review-rules) and make changes that impact your budgets. Manage license consumption by [reviewing licensing dashboards](/observe/dashboards/managed-dashboards#licensing-dashboards) to identify usage trends and avoid exceeding your organization's licensing limits. [Understand telemetry data limits](/administer/limits-licensing) for each telemetry type to ensure a performant experience when ingesting, querying, and creating resources. ## Create budgets After creating partitions and analyzing consumption, create and apply *budgets*, which are shaping policies you assign to a partition to safeguard against runaway usage and overspending. [Budgets](/control/consumption/budgeting) help track and manage growth to ensure your license consumption remains predictable. # Manage consumption Source: https://docs.chronosphere.io/control/consumption Learn how to create partitions and budgets for your telemetry data in Chronosphere Observability Platform and control the data you keep. Telemetry data provides your teams with more detailed context to help identify and fix the root causes of issues. However, not all telemetry data provides the same utility, and you don't want to consume a portion of your license with unnecessary data. To help control data usage, first create [*partitions*](/control/consumption/partitions), which are slices of your log data with distinct owners. For example, you can create partitions for individual services, teams, and environments to provide a consistent structure for attributing usage and costs to the proper owners. Partitions let you apply [*budgets*](/control/consumption/budgeting), which are shaping policies you can assign to a partition to safeguard against runaway usage and overspending. After defining partitions, use the [Consumption page](/control/consumption/analyze) to view persisted log data consumption against license capacity. You can then [shape log data](/control/shaping/shape-logs/parse-logs) to apply finer-grained control. ## How to control telemetry data To manage your telemetry data consumption, complete the following steps: 1. [Create partitions](/control/consumption/partitions). 2. [Analyze consumption](/control/consumption/analyze). 3. [Create budgets](/control/consumption/budgeting). 4. [Monitor budgets](/control/consumption/budgeting#next-steps). 5. [Shape telemetry data](/control/shaping/shape-logs). To delete data from your Observability Platform tenant, contact [Chronosphere Support](/support). # Analyze consumption Source: https://docs.chronosphere.io/control/consumption/analyze Analyze telemetry consumption across products, resources, and partitions to identify growth and unexpected spikes. After [creating partitions](/control/consumption/partitions), use the Consumption page to identify unexpected spikes, understand growth, and compare usage across partitions. The Consumption page displays consumption for the selected product, resource group, resource, and time range. For applicable global partition views, the page also displays contractual volume consumed and contract duration elapsed. The page includes budget thresholds and consumption statistics for child partitions. To compare telemetry rates with the system limits configured for your tenant, use the [Consumption System Limits dashboard](/observe/dashboards/managed-dashboards#consumption-system-limits). That dashboard is the source of truth for system limit thresholds, utilization, and data dropped by system limit enforcement. ## Analyze with the Consumption page To analyze data consumption in the Consumption page, in the navigation menu, click ** Go to Admin** and then select ** Control > Consumption**. ### Select consumption data Use the page controls to define the consumption data to analyze: 1. Select a **Product**, **Resource Group**, and **Resource**. The available resource groups and resources depend on the selected product. 2. Review the unit selector, if present. Select **Credits** or **Native units** when both options are enabled. The page uses credits when the selection includes multiple resources. 3. Select the page time range. The available ranges include **This month** and **Previous month**. The page time range controls which data the page retrieves. The **Time window** menu in the Consumption chart controls how the retrieved data is aggregated. ### Review partition statistics The partition summary is named for the selected partition. For example, the global partition displays the Global Partition summary. The summary includes the following statistics: * Use **Consumed** to view the total data consumed and persisted during the selected time range. For the global partition, this statistic also displays the percentage of contractual volume when contract details apply to the selected range. * Use **Dropped by budgets** to view the total data dropped by budget enforcement during the selected time range. * If **Dropped by system limits** appears, use it to view the total data dropped by system limit enforcement during the selected time range. Click **View** to open the [Consumption System Limits dashboard](/observe/dashboards/managed-dashboards#consumption-system-limits) for the same time range. * Use **Change over time** to view the percentage change in consumption compared to the preceding period. The comparison period depends on the selected time range. For the global partition, review **Contract duration elapsed** when the selected time range falls within the contract period. Archived and unpartitioned partitions display corresponding badges. ### Analyze consumption trends In the Consumption chart, use the **Time window** menu to select how the chart aggregates consumption: * Use **Per second** to detect spikes. * Use **Daily**, **Weekly**, and **Monthly (Cumulative)** to identify gradual growth or sustained overconsumption. * Use **Rolling 30 minutes**, **Rolling 1 hour**, **Rolling 3 hours**, **Rolling 1 day**, or **Rolling 7 days** to identify recurring changes over rolling periods. Use the **Group by** menu to select how the chart divides the data: * **Consumed vs dropped** compares persisted volume with dropped volume. * **Consumed by partition** identifies which partitions account for the most consumption. * **Consumed by product** compares consumption across telemetry products. * **Consumed by resource** compares the resources within the selected product. * **Dropped by budget** or **Dropped by reason** compares dropped volume. If **Dropped by reason** appears, use it to separate budget drops from system limit drops. * **Dropped by partition** identifies which partitions account for dropped data. If the chart includes [budget thresholds](/control/consumption/budgeting#threshold-types), dashed lines indicate the threshold values. Use **Show thresholds** to show or hide these lines. When **Go to Explorer** appears for log data, click it to investigate the underlying data in [Logs Explorer](/investigate/querying/query-logs). ### Review budgets The Budgets table displays the thresholds that apply to the selected partition. Use the following columns to review each threshold: * **Product** and **Resource Group** identify the data that the threshold governs. * **Time window** identifies the period over which the threshold is evaluated. * **Action** identifies whether the threshold sends a warning alert, sends a critical alert, or drops data. When an action has an associated monitor, click the action to open that monitor. * **Threshold** displays the received volume as a percentage and value relative to the threshold. The received volume includes consumed and dropped data. * **View** selects the threshold's product and resource group, resets the resource to **All**, restores the threshold's original unit, and selects the matching chart time window. The Budgets table header reports the selected partition's budget status as Ok, Alerting, or Dropping. A partition without a budget displays no status badge, and the table displays `No budget`. The table can also display a system limit as Per second (max consumption rate). The Consumption System Limits dashboard remains the source of truth for system limit thresholds and enforcement. The budget for the `global` partition might exceed its threshold while budgets for all child partitions remain within their thresholds. To make the global partition budget reflect growth trends more accurately, set thresholds on budgets for individual partitions. ### Compare child partitions The Child Partitions table compares the selected partition's immediate children. Search for a partition by name, or sort the table by the following columns: * **Order**: The order in which each child partition matches data. * **Partition**: The child partition name. Click a linked name to open that partition's Consumption page. Unpartitioned rows aren't linked. Archived and unpartitioned rows display corresponding badges. * **Consumed**: The partition's percentage of its parent's consumed volume and its absolute consumed value. * **Dropped**: The partition's percentage of its parent's dropped volume and its absolute dropped value. * **Change over time**: The percentage change in consumption compared to the preceding period. The column name reflects the comparison period. * **Current budget status**: The partition's current budget status. Partitions without a budget display `--`. Use this table to investigate partitions with growing consumption or dropped data. ## Next steps Based on your findings, refine your [partitions](/control/consumption/partitions) with more specific dimensions. For example, add child partitions to control smaller subsets of data. After analyzing your consumption and refining partitions, create [budgets](/control/consumption/budgeting) to apply shaping policies that provide guardrails against usage overages and overspending. # Telemetry data budgets Source: https://docs.chronosphere.io/control/consumption/budgeting Learn how to create and manage budgets and apply them to your data partitions. After [defining partitions](/control/consumption/partitions), [analyzing consumption](/control/consumption/analyze), and making any necessary adjustments, you can create *budgets*, which are shaping policies that you can attach to a partition. Budgets provide guardrails to combat runaway usage and overspending, while providing the flexibility to enforce accountability at the right level of ownership. Budgets include *thresholds* that define which actions to take when a threshold is exceeded. You can also set *priorities* to control the order in which data is dropped when a drop action is applied, with lower priority values dropped last. For example, a priority of `10` is dropped first, and a priority of `1` is dropped last. Priorities are evaluated in match order, and the first priority to match is applied. All other priorities are ignored. Use the `default_priority` field to set the priority on requests that don't match any defined priorities. ## Threshold types Budgets can include *thresholds*, which help manage both anomalous spikes in data and slow data growth over time, depending on the type. Use different threshold types depending on the use case. For each threshold, you can specify an action, such as sending an alert or dropping data. When you define an alert action on a threshold, such as sending a warning alert, Observability Platform generates an associated monitor in the **Chronosphere Managed** [collection](/administer/collections). Although configuring thresholds is optional, Chronosphere strongly recommends setting thresholds to protect budgets and avoid overspending. ### Anomalous spikes in real time To detect data spikes in real time and react accordingly, configure either an instant-rate threshold or a rolling threshold. These thresholds implement shorter time windows that are more responsive, but can result in more noise. Instant-rate thresholds limit the rate of incoming data in bytes per second, specified as fixed values. With an instant rate threshold, the configured action (such as sending an alert) triggers when consumption exceeds the configured rate threshold. For example, if a threshold is exceeded, Observability Platform can send an alert to notify the team who owns the budget for the offending partition. Alternatively, you can start dropping data when a threshold value is met so that the budget isn't exceeded, which would otherwise impact your persisted data limit. Instant-rate thresholds apply only to the `LOG_PERSISTED_BYTES` and `LOG_PROCESSED_BYTES` resource groups. Metric, trace, and credit-based resource groups don't support instant-rate thresholds. For those resource groups, use a rolling or fixed volume-based threshold instead. For alert actions on instant-rate thresholds, configure `instant_rate_sustain_secs` in the `alert_action_config` to control how long consumption must exceed the threshold before an alert triggers. The default is `0`, which triggers an alert immediately on any breach. Rolling thresholds help to detect and manage anomalous spikes in data over a short period, such as one to three hours. The available rolling threshold types are: * `ROLLING_30_MINUTE_VOLUME`: Rolling thirty-minute window. * `ROLLING_1_HOUR_VOLUME`: Rolling one-hour window. * `ROLLING_3_HOUR_VOLUME`: Rolling three-hour window. * `ROLLING_1_DAY_VOLUME`: Rolling one-day window. * `ROLLING_7_DAY_VOLUME`: Rolling seven-day window. ### Slow growth over time To help detect and manage gradual data growth over a longer period (which can be daily, weekly, or monthly), use fixed volume-based thresholds. These thresholds implement longer time windows that are less responsive, but are also less noisy than instant-rate or rolling thresholds. The available fixed volume threshold types are: * `DAILY_VOLUME`: Resets daily at `00:00:00` UTC. * `WEEKLY_VOLUME`: Resets weekly on Monday at `00:00:00` UTC. * `MONTHLY_VOLUME`: Resets on the first of each month at `00:00:00` UTC. Fixed volume-based thresholds set a limit over a fixed, non-overlapping period, such as one day. The configured action, such as dropping data, triggers when consumption over the specified time period exceeds the configured threshold. For example, dropping data on a daily basis if a budget for a service exceeds a fixed value threshold of a configured number of bytes. The `HOURLY_VOLUME` threshold type is deprecated, and the API rejects it when you create or update a budget. To set an hourly window, use `ROLLING_1_HOUR_VOLUME` instead. ### Drop data with thresholds When certain thresholds with a drop action are exceeded, Observability Platform drops data according to the length of the time window and the threshold type: * Instant-rate thresholds: Observability Platform accepts requests at a rate of up to the threshold value, and drops data when the rate is over the threshold in priority order. Data resumes when the incoming rate falls below the threshold. * Fixed volume-based thresholds: Observability Platform drops data until the end of the time window. For example, a daily volume-based threshold that's configured to drop data at 12:00 AM drops data until 11:59 PM. * Rolling thresholds: Observability Platform drops data until the data in the affected budget naturally ages out of the rolling window, and your tenant consumption is under the configured threshold. Avoid setting multiple volume-based thresholds with a drop action on the same budget. For example, setting a daily, weekly, and monthly volume-based threshold provides considerable overlap, and could result in unwanted noise, such as excessive alerts. The `DROP` action isn't supported for the trace resource groups (`TRACE_PROCESSED_BYTES`, `TRACE_PERSISTED_BYTES`, and `TRACE_ALL`) or for the cross-telemetry `ALL` resource group. For those resource groups, use an alert action instead. See [Create budgets](#create-budgets) for an explanation of configuring alert actions on budget thresholds. Each combination of resource group, action, and threshold type must be unique within a budget. For example, a budget can't contain two `DROP` thresholds that are both `INSTANT_RATE` for the `LOG_PERSISTED_BYTES` resource group. Two `DROP` and `INSTANT_RATE` thresholds are allowed when their resource groups differ. The `global` partition enforces the rate limits from your contract as system thresholds. You can't create a threshold on the `global` partition that duplicates a system-enforced combination of resource group, action, and threshold type. Set custom thresholds on child partitions instead. ## Budgeting order If you attach multiple budgets to every level of your partition hierarchy, a single request must pass through multiple thresholds before being accepted. Budgets are evaluated in order, from the most-specific partition that you created, to the least-specific partition (the global partition). If a budget drops a request, then data isn't counted in the received rates of any parent budgets. For example, consider the following partitions and budgets: * A partition for `global/team-a` has an `instant_rate` threshold of 2 MB per second. * A partition for `global/team-a/service-1` has an instant rate threshold of 1 MB per second. ```mermaid actions={false} theme={null} flowchart TD global --> team-a team-a --> service-1 ``` If the partition `global/team-a/service-1` receives a huge data spike of 1 GB per second, then 999 MB of that overage is dropped before it's counted against the budget for `global/team-a`. Therefore, the `global/team-a` partition receives only 1 MB per second, and won't be impacted by data drops. This budget hierarchy ensures that drop data from the `global/team-a/service-1` partition only, and not from `global/team-a`. The budget that's configured on the individual service ensures that the team budget isn't impacted. This behavior allows you to penalize bad actors and not impact the upstream partitions. ## View budgets Select from one of the following methods to view budgets for partitions. To view budgets for a partition: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Partitions and Budgets**. 2. Select a partition from the side navigation or the **Child partitions** table. 3. Click the **Budget** tab. The **Budget Thresholds** section displays configured thresholds, and the **Log priorities** section displays the priority order for drop actions. To use [Chronoctl](/tooling/chronoctl) to return all budgets, use the `chronoctl consumption-budgets list` command: ```shell theme={null} chronoctl consumption-budgets list ``` To filter for a specific budget, use the `chronoctl consumption-budgets read` command: ```shell /SLUG/ theme={null} chronoctl consumption-budgets read SLUG ``` Replace *`SLUG`* with the slug of the budget you want to display. To return a list of budgets with the Chronosphere API, use the [`ListConsumptionBudgets`](/tooling/api-info/definition/operations/ListConsumptionBudgets) endpoint. To return a single budget definition, use the [`ReadConsumptionBudget`](/tooling/api-info/definition/operations/ReadConsumptionBudget) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create budgets When creating a budget, complete the following actions: * Required: Include the slug path of the partition that the budget applies to. * Optional: Specify an alert notification policy to use for routing alerts. * Optional (but highly suggested): Define thresholds and actions to take if a threshold is exceeded. * Optional: Apply priorities for each of the query conditions that match. Configure alert actions before enforcing drops. Alerts provide visibility without impacting persisted data. For each alert you define on a threshold, Observability Platform generates an associated monitor in the **Chronosphere Managed** collection. To configure a generated monitor, specify metadata in the `alert_action_config` section of the budget definition. For example, you can add `annotations` that include links to runbooks, and specify `labels` that can include metadata like `team` and `environment`. These labels can be used for routing notifications, such as to specific teams to notify when an alert triggers. See the [Chronoctl](#chronoctl-budget-example) and [Terraform](#terraform-budget-example) examples for details. You can specify multiple alert actions for a single threshold. For example, define a warning alert at one value, and then a critical alert at a higher value. However, Observability Platform generates only one monitor for each [threshold type](#threshold-types) in a budget. This means that even if you define multiple alerts for a threshold type, such as an instant rate threshold, only one monitor is created. ### Prerequisites Before creating budgets, [define partitions](/control/consumption/partitions). Use the slug or ID of your partition to assign budgets. Because budgets use Chronosphere alerts to send notifications when thresholds are exceeded, you need to define [notification policies](/investigate/alerts/notifications/policies) to reference in your budgets. ### Create partition budgets Use one of the following methods to create budgets for partitions. Define the budget in Observability Platform, and then [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Partitions and Budgets**. 2. Select the partition you want to add a budget to. 3. Click the **Budget** tab. 4. In the **Budget Thresholds** section, click **Add threshold**. 5. For each threshold, select values for the **Product**, **Resource group**, **Time window**, and **Action** fields, and then enter a threshold value. After entering a threshold value, select a unit for the value. The available units depend on the selected resource group and time window. 6. Optional: In the **Log priorities** section, click **Add priority** to define drop-order priorities. For each priority, enter a value in the **Priority** field and a query in the **Filter** field. 7. Optional: In the **Notification policy** section, select a notification policy to route alerts raised by budget thresholds. 8. To save the budget and apply changes to the configuration, click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. To use [Chronoctl](/tooling/chronoctl) to create a budget, use the `chronoctl consumption-budgets create` command: ```shell theme={null} chronoctl consumption-budgets create ``` 1. Run the following command to generate a sample budget configuration you can use as a template: ```shell theme={null} chronoctl consumption-budgets scaffold ``` In the template, `kind: ConsumptionBudget` defines an individual budget. You can redirect the results (using the redirection operator `>`) to a file for editing. 2. With a completed definition, submit it with: ```shell /FILE_NAME/ theme={null} chronoctl consumption-budgets create -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. See the [Chronoctl example](#chronoctl-budget-example) for a completed budget resource. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. To create a budget with [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file and add the definition by using the `chronosphere_consumption_budget` type. 2. Enter a `consumption_config_id` for the budget. 3. Include the `notification_policy_id` that references the ID of the notification policy you want to use for routing alerts. 4. Include the `partition_slug_path` that references the slug of the partition you want to assign this budget to. 5. Define a `threshold` condition to send alerts. 6. Define a `threshold` condition to drop data. 7. Run this command to apply the changes: ```shell theme={null} terraform apply ``` See the [Terraform budget example](#terraform-budget-example) for a completed budget resource. To complete this action with the Chronosphere API, use the [`CreateConsumptionBudget`](/tooling/api-info/definition/operations/CreateConsumptionBudget) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Next steps After creating budgets, use the Consumption page to monitor and [analyze consumption](/control/consumption/analyze) for the budget you added to your partition: * Review relative consumption trends: In the **Consumption** panel group, use the dropdown menus to view consumption trends across different time periods to understand what's driving data growth. * Detect budgets approaching thresholds: Use the **Budgets** panel group to view data matching the selected partition against either the maximum consumption rate set in your contract for the global partition, or a custom threshold for custom partitions. * Observe drops from rate-limiting thresholds per partition: Use the **Consumed vs dropped** option in the **Group by** menu to view the rate at which data was dropped by a threshold for the selected partition, broken down by priority (if configured). ### Chronoctl budget example The following Chronoctl example adds a budget for the Ordering Service, which is included in the [partition hierarchy](/control/consumption/partitions#partition-hierarchy) diagram. See the [Chronoctl example](/control/consumption/partitions#chronoctl-partition-example) that implements a partition for this service. This budget is scoped to the partition `global/ordering-team/ordering-service`, and defines the following thresholds: * An alert warning (`ALERT_WARN`) with a threshold of 4,500 bytes per second. * A critical warning (`ALERT_CRITICAL`) with a threshold of 5,000 bytes per second. * A drop action (`DROP`) with a threshold of 5,000 bytes per second. * A drop action (`DROP`) for daily volume, with a threshold of 909,997 bytes. Each `priority` controls which data gets dropped first when the `DROP` action triggers. Higher priority values are dropped first, and lower priority values are dropped last: * The query `"env = 'dev'"` has a priority of `10`, so it's dropped first. * The query `"env = 'staging'"` has a priority of `5`, so it's dropped next. * The query `"env = 'prod'"` has a priority of `1`, so it's dropped last. ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: ConsumptionBudget spec: slug: "ordering-service-logs" name: "Ordering Service Logs" partition_slug_path: "global/ordering-team/ordering-service" alert_action_config: annotations: runbook: "https://wiki.example.com/runbooks/log-budget-alerts" team: "ordering-engineering" description: "Custom description for log budget alerts" labels: severity: "high" environment: "production" service: "ordering-service" instant_rate_sustain_secs: 60 priorities: - filters: - log_filter: query: "env = 'dev'" priority: 10 - filters: - log_filter: query: "env = 'staging'" priority: 5 - filters: - log_filter: query: "env = 'prod'" priority: 1 thresholds: - action: ALERT_WARN instant_rate: fixed_value_per_sec: 4500 type: INSTANT_RATE resource_group: LOG_PERSISTED_BYTES unit: NATIVE - action: ALERT_CRITICAL instant_rate: fixed_value_per_sec: 5000 type: INSTANT_RATE resource_group: LOG_PERSISTED_BYTES unit: NATIVE - action: DROP instant_rate: fixed_value_per_sec: 5000 type: INSTANT_RATE resource_group: LOG_PERSISTED_BYTES unit: NATIVE - action: DROP volume: fixed_value: 909997 type: DAILY_VOLUME resource_group: LOG_PERSISTED_BYTES unit: NATIVE notification_policy_slug: "budget-alerts-slack" ``` ### Terraform budget example The following Terraform example adds a budget for the Ordering Service, which is included in the [partition hierarchy](/control/consumption/partitions#partition-hierarchy) diagram. See the [Terraform example](/control/consumption/partitions#terraform-partition-example) that implements a partition for this service. This budget is scoped to the partition `global/ordering-team/ordering-service`, and defines the following thresholds: * An alert warning (`ALERT_WARN`) with a threshold of 4,500 bytes per second. * A critical warning (`ALERT_CRITICAL`) with a threshold of 5,000 bytes per second. * A drop action (`DROP`) with a threshold of 5,000 bytes per second. * A drop action (`DROP`) for daily volume, with a threshold of 909,997 bytes. Each `priority` controls which data gets dropped first when the `DROP` action triggers. Higher priority values are dropped first, and lower priority values are dropped last: * The query `"env = 'dev'"` has a priority of `10`, so it's dropped first. * The query `"env = 'staging'"` has a priority of `5`, so it's dropped next. * The query `"env = 'prod'"` has a priority of `1`, so it's dropped last. The `notification_policy_id` references a notification policy created in Terraform. To reference a notification policy created in Observability Platform, enclose the value in double quotes. ```terraform expandable Terraform example icon="square-terminal" theme={null} resource "chronosphere_consumption_budget" "ordering_service_budget" { consumption_config_id = chronosphere_consumption_config.example.id name = "Ordering Service Logs" slug = "ordering-service-logs" notification_policy_id = chronosphere_notification_policy.budget_alerts_slack.id partition_slug_path = "global/ordering-team/ordering-service" # Alert action configuration alert_action_config { # Additional annotations to set on generated monitors # These override settings on default annotations if they match default keys annotations = { "runbook" = "https://wiki.example.com/runbooks/log-budget-alerts" "team" = "ordering-engineering" "description" = "Custom description for log budget alerts" } # Additional labels for notification routing labels = { "severity" = "high" "environment" = "production" "service" = "ordering-service" } # Determines how long instant rate consumption must sustain above the threshold # before triggering an alert # Default is 0 (any consumption over threshold triggers immediately) instant_rate_sustain_secs = 60 } # Threshold configurations threshold { action = "DROP" type = "INSTANT_RATE" resource_group = "LOG_PERSISTED_BYTES" unit = "NATIVE" instant_rate { fixed_value_per_sec = 5000 } } threshold { action = "ALERT_WARN" type = "INSTANT_RATE" resource_group = "LOG_PERSISTED_BYTES" unit = "NATIVE" instant_rate { fixed_value_per_sec = 4500 } } threshold { action = "ALERT_CRITICAL" type = "INSTANT_RATE" resource_group = "LOG_PERSISTED_BYTES" unit = "NATIVE" instant_rate { fixed_value_per_sec = 5000 } } threshold { action = "DROP" type = "DAILY_VOLUME" resource_group = "LOG_PERSISTED_BYTES" unit = "NATIVE" volume { fixed_value = 909997 } } # Priorities determine the drop order # The default priority applies to requests that don't match any priority filter. # If unset, the priority defaults to 10 (dropped first). default_priority = 5 priority { priority = 1 filter { log_filter { query = "env = 'prod'" } } } priority { priority = 5 filter { log_filter { query = "env = 'staging'" } } } priority { priority = 10 filter { log_filter { query = "env = 'dev'" } } } } ``` ## Update budgets Select from one of the following methods to update budgets for available partitions. Any changes to the `notification_policy` or `alert_action_config` are automatically updated in the monitor for the associated threshold. If you remove an existing threshold from a budget that includes an alert action, the monitor that's generated automatically from that alert action is deleted from Observability Platform. To edit a budget in Observability Platform: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Partitions and Budgets**. 2. Select the partition whose budget you want to update. 3. Click the **Budget** tab. 4. To edit a threshold, click the actions menu on the threshold row and select **Edit**. Update the values inline. 5. To reorder priorities, drag and drop priority rows to change their match order. 6. To change the notification policy, select a different policy from the **Notification policy** dropdown. 7. To save changes to the configuration, click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. To update budgets with [Chronoctl](/tooling/chronoctl), use the `consumption-budgets update` command: ```shell theme={null} chronoctl consumption-budgets update ``` To edit a budget using [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file that updates the resource's existing properties. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`UpdateConsumptionBudget`](/tooling/api-info/definition/operations/UpdateConsumptionBudget) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete budgets Select from one of the following methods to delete budgets for partitions. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To delete individual thresholds or priorities from a budget: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Partitions and Budgets**. 2. Select the partition whose budget items you want to remove. 3. Click the **Budget** tab. 4. To delete a threshold, click the actions menu on the threshold row and select **Delete**. 5. To delete a priority, click the actions menu on the priority row and select **Delete**, and then confirm the deletion. 6. To apply the deletion, click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. To delete budgets with [Chronoctl](/tooling/chronoctl), use the `consumption-budgets delete` command: ```shell theme={null} chronoctl consumption-budgets delete SLUG ``` Replace *`SLUG`* with the slug of the budget you want to delete. To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteConsumptionBudget`](/tooling/api-info/definition/operations/DeleteConsumptionBudget) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # Partitions Source: https://docs.chronosphere.io/control/consumption/partitions Use partitions in Chronosphere Observability Platform to create slices of your data with meaningful dimensions and distinct owners. To effectively manage your telemetry data, you need to separate, or partition data into meaningful slices that map to your business. *Partitions* provide a consistent structure for attributing usage and costs to the appropriate owners in your organization so they can isolate and independently control their parts of the business. ## Partition hierarchy Partitions are hierarchical, which lets you model the ownership structure of your organization. A global partition is created by default, and captures all consumption. Individual teams can create distinct partitions, with child partitions for individual services and environments. All created partitions are children of the global partition, and each created partition has a default partition to collect any traffic not explicitly captured by other partitions. Default partitions are created automatically, and ensure that all consumption is accounted for at every level and sums correctly into parent partitions. The following diagram depicts the hierarchy of partitions stemming from the `global` partition. Two created partitions are children of the `global` partition: one for the `ordering_team` and another for the `auth-team`. Each of those partitions have child partitions for each team's individual service, with additional child partitions for `prod`, `staging`, and `dev` environments. ```mermaid actions={false} theme={null} --- title: Partition tree --- flowchart TD accTitle: Partitions stemming from the global partition accDescr: The diagram shows a global partition, with three child partitions. One each for the ordering team, auth team, and a default partition. The ordering team and auth team partitions have child partitions for their individual services, which have their own partitions for different environments. global[global] global --> ordering_team[ordering-team] ordering_team --> ordering_service[ordering-service] ordering_service --> ordering_prod[prod] ordering_service --> ordering_staging[staging] ordering_service --> ordering_dev[dev] ordering_service --> ordering_default[default] ordering_team --> ordering_team_default[default] global --> auth_team[auth-team] auth_team --> auth_service[auth-service] auth_service --> auth_prod[prod] auth_service --> auth_staging[staging] auth_service --> auth_dev[dev] auth_service --> auth_default[default] auth_team --> auth_team_default[default] global --> global_default[default] classDef green fill:#28a561,stroke-width:2px,stroke:#28a561,color:#FFFFFF,font-weight:bold,underline; classDef purple fill:#705DA0,stroke-width:2px,stroke:#705DA0,color:#FFFFFF,font-weight:bold,underline; classDef orange fill:#ff9d1c,stroke-width:2px,stroke:#ff9d1c,color:#FFFFFF,font-weight:bold,underline; class ordering_team,ordering_service,ordering_dev,ordering_staging,ordering_prod green class auth_team,auth_service,auth_prod,auth_staging,auth_dev purple class global_default,ordering_team_default,ordering_default,auth_team_default,auth_default orange ``` ## Filters Each partition uses one or more filters to determine which telemetry data the partition captures. A request must match every filter to be assigned to the partition. Each filter uses one of the following operators: * `IN`: the request must match at least one of the filter's conditions. * `NOT_IN`: the request must not match any of the filter's conditions. A partition with multiple `IN` or `NOT_IN` filters save each condition into a single `IN` and `NOT_IN` operator collection. Each condition matches exactly one telemetry type through one of the following fields: * `log_filter`: matches log data with a log query. * `metric_filters`: matches metric data by label. A metric must match every label filter in the condition. Label values support glob patterns, such as `service:{svc1,svc2}` to match alternatives. * `trace_span_filters`: matches trace data at the span level. A span must satisfy every span filter in the condition. ### Trace span filters A trace span filter matches spans on one or more of the following fields. To match alternatives, use an `IN` string filter or separate conditions. * `service`: the service of the span. * `operation`: the operation of the span. * `parent_service`: the service of the span's parent span. Root spans don't match. * `parent_operation`: the operation of the span's parent span. Root spans don't match. * `duration`: the duration of the span. An omitted or zero bound is unbounded on that side. * `error`: the error status of the span. * `tags`: the tags of the span. A numeric matcher also compares numeric-looking string tag values. * `is_root_span`: whether the span is the root span of its trace. ## View partitions Select from the following methods to view and filter available partitions. To view partitions: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Partitions and Budgets**. The global partition and any defined child partitions display. 2. To view the definition for a specific partition, select it from the **Child partitions** table, or from the side navigation. The child partition's definition displays in the **Details** tab. 3. With a child partition selected, click the **Budget** tab to view the defined [budget thresholds](/control/consumption/budgeting) attached to that partition. To use [Chronoctl](/tooling/chronoctl) to return all partitions, use the `chronoctl consumption-config read` command: ```shell theme={null} chronoctl consumption-config read ``` To complete this action with the Chronosphere API, use the [`ReadConsumptionConfig`](/tooling/api-info/definition/operations/ReadConsumptionConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create partitions Select from the following methods to create partitions. Each partition can have up to five levels of nesting. Partitions are applied in match order as listed in your configuration file. If data matches multiple partitions, Chronosphere Observability Platform applies the first partition that the data matches in the configuration file. The total number of partitions is limited, including the default partitions that Observability Platform generates automatically for each non-empty partition list. If you reach the limit, consolidate partitions or contact [Chronosphere Support](/support). Define the partition definition in Observability Platform, and then [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Partitions and Budgets**. 2. Click **New partition**, and then enter a name for your partition. 3. In the side navigation, select your new partition to configure it. 4. If you want to use a different display name or slug for the new partition, enter new values. After applying the partition configuration, the slug value can't be changed. 5. Select the parent partition for the new partition to belong to. 6. Define filters, which determine the data that belongs to this partition. For each filter, select **In** or **Not in** from the **Operator** menu to include or exclude matching data, and then enter the **Conditions**. The condition format depends on the telemetry type: * Logs: a log query. * Metrics: label-value pairs. * Traces: span fields, such as service or operation. For how the **In** and **Not in** operators combine, see [Filters](#filters). For examples of filters, see the [Chronoctl example](#chronoctl-partition-example) and the [Terraform example](#terraform-partition-example). 7. To create child partitions in the new partition, in the **Child partitions** section, click **New partition**. Partitions support a maximum nesting depth of five levels from the global partition. 8. To save the new partition and apply changes to the configuration, click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. After applying the changes with the Code Config tool, the new partitions displays in the list of available partitions. To use [Chronoctl](/tooling/chronoctl) to create partitions, use the `chronoctl consumption-config create` command: ```shell theme={null} chronoctl consumption-config create ``` 1. Run the following command to generate a sample partition configuration you can use as a template: ```shell theme={null} chronoctl consumption-config scaffold ``` In the template, `kind: ConsumptionConfig` defines an individual partition. 2. With a completed definition, submit it with: ```shell /FILE_NAME/ theme={null} chronoctl consumption-config create -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. See the [Chronoctl partition example](#chronoctl-partition-example) for a completed partition definition. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. To create a partition with [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file and add the definition by using the `chronosphere_consumption_config` type, followed by a name in a resource declaration. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` See the [Terraform partition example](#terraform-partition-example) for a completed partition resource. To complete this action with the Chronosphere API, use the [`CreateConsumptionConfig`](/tooling/api-info/definition/operations/CreateConsumptionConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Next steps After creating a partition, use the [Consumption](/control/consumption/analyze) page to view usage across partitions to understand what's driving growth and detect unwanted spikes. From the **Partitions and Budgets** page in Observability Platform, click **View in Consumption** to open the **Consumption** page. After analyzing consumption, create [budgets](/control/consumption/budgeting) and assign them to each of your partitions. ### Chronoctl partition example The following Chronoctl example defines two partitions at the team level, each with a child service partition and environment sub-partitions for production, staging, and development environments. Each partition uses conditions for logs, metrics, and traces so that data from all three telemetry types is attributed to the correct owner. The `Ordering Team` partition matches services with a regular expression log query, a glob-based metric filter, and an `IN` trace span filter. The `Auth Team` partition adds a `NOT_IN` filter to exclude load-test traffic from its consumption totals. ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: ConsumptionConfig spec: partitions: - name: "Ordering Team" slug: "ordering-team" filters: - operator: "IN" conditions: - log_filter: query: "service =~ 'ordering|order-processor'" - metric_filters: - name: "service" value_glob: "ordering*" - trace_span_filters: - service: match: "IN" in_values: - "ordering" - "order-processor" partitions: - name: "Ordering Service" slug: "ordering-service" filters: - operator: "IN" conditions: - log_filter: query: "service = 'ordering'" - metric_filters: - name: "service" value_glob: "ordering" - trace_span_filters: - service: match: "EXACT" value: "ordering" partitions: - name: "Prod" slug: "prod" filters: - operator: "IN" conditions: - log_filter: query: "env = 'prod'" - metric_filters: - name: "env" value_glob: "prod" - name: "Staging" slug: "staging" filters: - operator: "IN" conditions: - log_filter: query: "env = 'staging'" - metric_filters: - name: "env" value_glob: "staging" - name: "Dev" slug: "dev" filters: - operator: "IN" conditions: - log_filter: query: "env = 'dev'" - metric_filters: - name: "env" value_glob: "dev" - name: "Auth Team" slug: "auth-team" filters: - operator: "IN" conditions: - log_filter: query: "service =~ 'auth|auth-proxy'" - metric_filters: - name: "service" value_glob: "auth*" - trace_span_filters: - service: match: "IN" in_values: - "auth" - "auth-proxy" - operator: "NOT_IN" conditions: - log_filter: query: "env = 'load-test'" - metric_filters: - name: "env" value_glob: "load-test" partitions: - name: "Auth Service" slug: "auth-service" filters: - operator: "IN" conditions: - log_filter: query: "service = 'auth'" - metric_filters: - name: "service" value_glob: "auth" - trace_span_filters: - service: match: "EXACT" value: "auth" partitions: - name: "Prod" slug: "prod" filters: - operator: "IN" conditions: - log_filter: query: "env = 'prod'" - metric_filters: - name: "env" value_glob: "prod" - name: "Staging" slug: "staging" filters: - operator: "IN" conditions: - log_filter: query: "env = 'staging'" - metric_filters: - name: "env" value_glob: "staging" - name: "Dev" slug: "dev" filters: - operator: "IN" conditions: - log_filter: query: "env = 'dev'" - metric_filters: - name: "env" value_glob: "dev" ``` ### Terraform partition example The following Terraform example defines the same structure as the [Chronoctl example](#chronoctl-partition-example), which includes two team partitions with nested service and environment partitions. Each filter block includes separate conditions for `log_filter`, `metric_filter`, and `trace_span_filters` so that logs, metrics, and traces are all routed to the matching partition. The `Auth Team` partition demonstrates combining an `IN` filter with a `NOT_IN` filter to include `auth` services while excluding `load-test` environment data. ```terraform expandable Terraform example icon="square-terminal" theme={null} resource "chronosphere_consumption_config" "example" { partition { name = "Ordering Team" slug = "ordering-team" filter { operator = "IN" condition { log_filter { query = "service =~ 'ordering|order-processor'" } } condition { metric_filter { name = "service" value_glob = "ordering*" } } condition { trace_span_filters { service { match = "IN" in_values = ["ordering", "order-processor"] } } } } partition { name = "Ordering Service" slug = "ordering-service" filter { operator = "IN" condition { log_filter { query = "service = 'ordering'" } } condition { metric_filter { name = "service" value_glob = "ordering" } } condition { trace_span_filters { service { match = "EXACT" value = "ordering" } } } } partition { name = "Prod" slug = "prod" filter { operator = "IN" condition { log_filter { query = "env = 'prod'" } } condition { metric_filter { name = "env" value_glob = "prod" } } } } partition { name = "Staging" slug = "staging" filter { operator = "IN" condition { log_filter { query = "env = 'staging'" } } condition { metric_filter { name = "env" value_glob = "staging" } } } } partition { name = "Dev" slug = "dev" filter { operator = "IN" condition { log_filter { query = "env = 'dev'" } } condition { metric_filter { name = "env" value_glob = "dev" } } } } } } partition { name = "Auth Team" slug = "auth-team" filter { operator = "IN" condition { log_filter { query = "service =~ 'auth|auth-proxy'" } } condition { metric_filter { name = "service" value_glob = "auth*" } } condition { trace_span_filters { service { match = "IN" in_values = ["auth", "auth-proxy"] } } } } filter { operator = "NOT_IN" condition { log_filter { query = "env = 'load-test'" } } condition { metric_filter { name = "env" value_glob = "load-test" } } } partition { name = "Auth Service" slug = "auth-service" filter { operator = "IN" condition { log_filter { query = "service = 'auth'" } } condition { metric_filter { name = "service" value_glob = "auth" } } condition { trace_span_filters { service { match = "EXACT" value = "auth" } } } } partition { name = "Prod" slug = "prod" filter { operator = "IN" condition { log_filter { query = "env = 'prod'" } } condition { metric_filter { name = "env" value_glob = "prod" } } } } partition { name = "Staging" slug = "staging" filter { operator = "IN" condition { log_filter { query = "env = 'staging'" } } condition { metric_filter { name = "env" value_glob = "staging" } } } } partition { name = "Dev" slug = "dev" filter { operator = "IN" condition { log_filter { query = "env = 'dev'" } } condition { metric_filter { name = "env" value_glob = "dev" } } } } } } } ``` ## Update partitions Select from the following methods to update partitions. To update partitions: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Partitions and Budgets**. 2. In the sidebar navigation, select the partition you want to update. Alternatively, in the **Child partitions** table, click the name of the partition, or click the three vertical dots icon in the row of the partition you want to update and then click **Edit**. 3. Make changes to the partition in the definition pane. 4. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your changes. To update partitions with [Chronoctl](/tooling/chronoctl), use the `consumption-config update` command: ```shell theme={null} chronoctl consumption-config update ``` 1. Update the partition definition file. 2. Run the following command to submit the changes: ```shell /FILE_NAME/ theme={null} chronoctl consumption-config update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. To edit partitions using [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file that updates the resource's existing properties. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`UpdateConsumptionConfig`](/tooling/api-info/definition/operations/UpdateConsumptionConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete partitions Select from the following methods to delete partitions. To delete partitions with attached budgets, you must first [delete attached budgets](/control/consumption/budgeting#delete-budgets), and then delete the partition. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To delete partitions: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Partitions and Budgets**. 2. In the sidebar navigation, select the partition you want to delete. Alternatively, in the **Child partitions** table, click the three vertical dots icon in the row of the partition you want to delete and then click **Delete**. 3. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your changes. To delete a partition with [Chronoctl](/tooling/chronoctl), use the `chronoctl consumption-config delete` command: ```shell theme={null} chronoctl consumption-config delete ``` To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteConsumptionConfig`](/tooling/api-info/definition/operations/DeleteConsumptionConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # Shape and sample your data Source: https://docs.chronosphere.io/control/shaping Learn how to use shaping rules to take control of your telemetry data. To take control of your telemetry data, Chronosphere Observability Platform provides multiple tools you can use to identify and eliminate data that isn't used and doesn't deliver value to your business. Use the Chronosphere Control Plane to reduce the amount of data retained in your system over time. Create *shaping rules* to drop data before it reaches Observability Platform, aggregate and rewrite data into more manageable and usable statistics, and alias expressions to improve data queries and references. ## Shape metric data Use the Chronosphere Control Plane to [shape your metric data](/control/shaping/shape-metrics) and reduce the amount of data you persist. Manage your telemetry data with tools that help you shape that traffic, such as creating drop rules, quotas, and aggregation rules. ## Sample trace data Create [trace datasets](/control/shaping/sample-traces/datasets) that map tracing license consumption to relevant business units in your organization. Datasets let you track and measure trace data usage over time before you implement proactive sampling strategies, whereas head and tail sampling alone require you to measure and implement together. You can then apply [behaviors](/control/shaping/sample-traces/behaviors) to set sampling rules for your datasets without having to write, coordinate, and manage head and tail sampling rules individually. ## Control log data Understanding your license consumption helps identify where you're spending the most money on your log data, and which services or operations are consuming the most of your license capacity. [Control log data](/control/shaping/shape-logs) to determine what portion of your license is allocated to the most important data. ## Review impact After creating rules that shape and sample your data, use the tools that Observability Platform provides to [review rule impact](/control/shaping/review-impact) and ensure the rules you created have the intended outcome. # Review shaping rules impact Source: https://docs.chronosphere.io/control/shaping/review-impact Learn how to review the impact of your shaping rules to better apply controls to your telemetry data. After [creating shaping rules](/control/shaping), you want to understand their impact to ensure they have the intended outcome. Chronosphere Observability Platform provides several tools to understand the impact of the shaping rules you create. ## Review metric shaping rules Use the [Aggregation Rules UI](/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules) to understand your existing metric shaping rules and how they affect your environment. Use the [shaping impact preview](/control/shaping/shape-metrics/reduce-cardinality/shaping-impact) to preview the impact of a rule on your overall system, which helps prevent breaking changes and ensure the rules you create operate as expected. You can then use the [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer) to review metrics with a **Data phase** filter that shows what data was dropped, matched, or rejected by specific rule types. Periodically review the [Recommendations](/control/shaping/shape-metrics/reduce-cardinality/recommendations) page to help identify metrics and labels with no usage or utility over the past 30 days. You can then create drop rules and rollup rules based on the recommendations. If your organization uses capacity pricing, you can define metrics [quotas and pools](/control/shaping/shape-metrics/quotas/manage-pools) to assign specific percentages of your total persisted writes limit to pools of metrics. If you identify data with no business value, apply [relabel rules](/investigate/querying/metrics/derived-telemetry#differences-between-relabeling-and-deriving) in your collector to adjust which metrics you send from the client side to Observability Platform. ## Review log control rules When [creating a log shaping rule](/control/shaping/shape-logs), enter a filter to display a preview of the log data that the filter matches. This preview lets you confirm that the control rule targets the intended logs before applying the control rule. After creating log shaping rules, use the [Logs Usage](/investigate/analyze/logs-usage) page to review and understand frequent patterns in your log data. Use this information to identify high-volume, low-utility patterns, and confirm that a pattern's volume has changed after applying a control rule. You can then create a control rule for a pattern from the **Logs Usage** page with a defined filter, which ensures you're matching the correct pattern. ## Review trace control rules Use the [Trace Control Plane](/control/shaping/sample-traces) to both create and review the control mechanisms for trace data. After creating datasets, you can view the total processed and persisted bytes for a selected period to understand the data consumption. Before assigning behaviors, [run a preview](/control/shaping/sample-traces/behaviors#preview-behaviors) against a specific dataset to understand how it impacts data volume before shaping the data. Select a behavior to view head and tail sampling statistics so you can understand how the behavior affects traces that impact multiple datasets. After creating a [trace metric](/control/shaping/sample-traces/trace-metrics), you can pivot to view similar traces or view similar logs to cross-reference what data the metric is capturing. # Sample your traces Source: https://docs.chronosphere.io/control/shaping/sample-traces Use sampling in Chronosphere Observability Platform to help control costs and maximize the usefulness of your tracing data. Distributed traces provide an additional layer of context for solving problems across complex systems that include hundreds or thousands of microservices. However, you want to ensure you're only ingesting tracing data that's relevant and valuable. To help control costs and maximize the usefulness of your tracing data, you can use the Trace Control Plane to narrow your focus to only a representative sample of your data and drop everything else. To access the Trace Control Plane, in the navigation menu select ** Control > Trace Control Plane**. ## Head and tail sampling *Head sampling* and *tail sampling* are the best-known strategies for sampling trace data * [Head sampling](/control/shaping/sample-traces/head-sampling) is a more blunt strategy that seeks to make a sampling decision as early as possible. Head sampling evaluates only a defined percentage of traces to take a representative sample of whole traces. * [Tail sampling](/control/shaping/sample-traces/tail-sampling) is more fine-grained, and evaluates every trace after assembling all spans. Tail sampling rules can consider request outcomes, such as whether a request succeeded or how long it took to complete, which isn't possible with head sampling. Creating and managing head and tail sampling rules can be challenging to ensure you're discarding and keeping the most impactful data. To simplify this process and decrease the learning curve of sampling, Chronosphere developed two concepts to group, track, and apply sampling rules: [datasets](#datasets) and [behaviors](#behaviors). ## Datasets Create [*datasets*](/control/shaping/sample-traces/datasets) to map sets of traces to named groups relevant to your organization so you can track processed and persisted bytes for those groups over time. Datasets don't impact your license consumption, so you can experiment with creating datasets to understand your license usage and make changes as needed without consuming a portion of your license. With datasets in place, you can then apply behaviors to your datasets. ## Behaviors After creating datasets for individual business units, you can apply [*behaviors*](/control/shaping/sample-traces/behaviors) to your datasets to set sampling rates without needing to write and manage large sets of fine-grained sampling rules. You set a baseline behavior that implements data-driven best practices with default parameters. You can modify those parameters based on the needs of your organization. For example, modify the defined criteria to drop low-value traces as quickly as possible and keep high-value traces at a specified rate for one or more datasets from a single behavior. You can also set a behavior to allow (sample at 100%) or deny (sample at 0%) all traces for a specific period. For example, set an allow behavior when you need to increase the amount of high-fidelity data during a deploy, or when debugging issues. Alternatively, set a deny behavior when you want to decrease the amount of noisy or spam traces to keep your budget spend within limits. ## Get started with sampling Complete the following steps to get started with trace sampling in Observability Platform. 1. [Instrument head sampling](/control/shaping/sample-traces/head-sampling#configure-head-sampling), which is a prerequisite for using behaviors. Head sampling drives total trace volume for each root service and operation. Use the Trace Control plane to manage head sampling as part of behaviors. 2. Optional: Create head and tail sampling rules: * [Create head sampling rules](/control/shaping/sample-traces/head-sampling#create-head-sampling-rules) * [Create tail sampling rules](/control/shaping/sample-traces/tail-sampling#create-tail-sampling-rules) Create these rules if you don't want to use behaviors for sampling management, or if you already have head and tail sampling rules you want to migrate to and manage within Observability Platform. 3. [Use the incomplete traces dataset](/control/shaping/sample-traces/datasets#identify-incomplete-traces) to identify incomplete traces. Incomplete traces are those where one or more spans lacks a parent span, like from a lack of instrumentation. 4. Recommended: Improve local instrumentation to ensure a higher volume of complete traces before creating independent datasets. 5. [Create a dataset](/control/shaping/sample-traces/datasets#create-datasets) per team or per environment to track the exact volume of trace data for that team or environment over time. 6. [Assign a baseline behavior](/control/shaping/sample-traces/behaviors#get-started-with-behaviors) to your datasets to ensure that you capture all of your most meaningful traces (such as slow traces and error traces) for incident response purposes. Use the baseline behavior to optionally capture fewer of your lower-priority traces (such as fast traces and successful traces) to obtain a system baseline. As you learn more about your trace data, you can [edit facets of the baseline behavior](/control/shaping/sample-traces/behaviors#customize-your-sampling-strategy) to modify your sampling strategy and more clearly define which traces to drop and which to keep. In most cases, you want to assertively drop less interesting, lower-value traces and keep more interesting, higher-value traces. In the Observability Platform app, you can modify the baseline sampling strategy and apply it across one or more datasets. # Use trace behaviors with datasets Source: https://docs.chronosphere.io/control/shaping/sample-traces/behaviors Use behaviors in Chronosphere Observability Platform to set sampling rules for your trace datasets without writing fine-grained sampling rules. Controlling the tracing data you process and persist is necessary to manage your license consumption. After creating [datasets](/control/shaping/sample-traces/datasets), you can apply trace *behaviors* to set sampling rates for your datasets without needing to write fine-grained sampling rules. Although datasets are the underpinning of behaviors, the behaviors themselves let you change the sampling rates of one or more datasets from either the Chronosphere Observability Platform app or by using Chronoctl. ## Get started with behaviors Create the right datasets for your organization and then use behaviors to manage the data that those datasets generate. Use the built-in [behavior types](#trace-behavior-types) to control sampling rates and apply them across datasets. Observability Platform includes an unassigned dataset you can use for experimentation. The baseline behavior is mapped to the unassigned dataset by default, which lets you start sending traces to Observability Platform with pre-configured sampling rules based on best practices. As you learn about how the unassigned dataset generates data, create additional datasets for each of your individual services. As you learn more about your trace data, you can [edit facets of the baseline behavior](#customize-your-sampling-strategy) to modify your sampling strategy and more clearly define which traces to drop and which to keep. In most cases, you want to assertively drop less interesting, lower-value traces and keep more interesting, higher-value traces. In the Observability Platform app, you can quickly modify the baseline sampling strategy and apply it across one or more datasets. Trace behaviors implement head sampling rules by default, which means you need only set the percentage at which you want to keep processed data. For example, if you set the head sampling rate to 100%, you're keeping 100% of traces. If you don't set a behavior for a dataset, Observability Platform uses defined [tail sampling](/control/shaping/sample-traces/tail-sampling) rules. However, you can use trace datasets and behaviors without needing to write tail sampling rules. To specify a sample rate for head sampling in the baseline behavior, you must instrument [head sampling](/control/shaping/sample-traces/head-sampling) first. Chronosphere supports the OpenTelemetry JaegerRemoteSampler head sampling standard. Explicitly-defined [head sampling rules](/control/shaping/sample-traces/head-sampling#create-head-sampling-rules) take precedence over head sampling rules defined by a behavior. To use behavioral head sampling, [edit the baseline behavior](#edit-the-baseline-behavior) to define a head sampling strategy, and then remove your existing head sampling rules. Behaviors are part of the Trace Control Plane, which also includes datasets and head and tail sampling rules. You need administrative access to use the Trace Control Plane. ## Trace behavior types Trace behaviors can be one of the following types: * **Baseline behavior**: Sample data in your datasets using data-driven best practices. Select which facets to apply to your low-value and high-value trace data, and modify the criteria as you learn more about the characteristics of your trace data. The baseline behavior is mostly proactive, and helps to identify what trace data to bring into Observability Platform and what data to drop. * **Allow behavior**: Sample your data at 100% to allow all traces. The allow behavior can be both proactive and reactive. For example, you might want more high-fidelity data during a deploy to catch any issues (proactive), or allow all traces from a specific service or operation when debugging issues (reactive). * **Deny behavior**: Sample your data at 0% to block all traces. The deny behavior can be both proactive and reactive. For example: * Proactive: drop all traces from a service because the data isn't needed. * Reactive: stop traces from a dataset that's currently generating too much data so you don't exceed your license limit. ## View behaviors You can view and filter available trace behaviors using Observability Platform, and return the trace behavior definition using Chronoctl. To view trace behaviors: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. Select the **Behaviors** tab to view the baseline, allow, and deny behaviors, including which datasets each behavior type currently applies to. The dataset match criteria displays the dataset definition the behavior operates on. 3. Select the individual behavior you want to view details for: * **Allow**: Displays the assignment history for the datasets where the allow behavior is currently active, and the datasets where this behavior ended. * **Deny**: Displays the assignment history for the datasets where the deny behavior is currently active, and the datasets where this behavior ended. * **Baseline**: Displays head and tail sampling statistics for the baseline behavior when it was active for any datasets. The assignment history displays the datasets where the baseline behavior is currently active, and the datasets where this behavior ended. You can [edit the baseline behavior](#customize-your-sampling-strategy) to change the tail sampling methodology and modify the facets that define which traces to drop and keep. To use [Chronoctl](/tooling/chronoctl) to return the trace behavior definition, use the `chronoctl trace-behavior-config read` command: ```shell theme={null} chronoctl trace-behavior-config read ``` The definition includes the main behavior and override behavior assignments for each of your datasets, and also displays the dataset shaping order. See the [Manage assigned behaviors](#manage-assigned-behaviors) section of this page for more information about assigning behaviors to datasets. See the [Chronoctl behavior example](/control/shaping/sample-traces/datasets#chronoctl-behavior-example) for a completed behavior definition. To complete this action with the Chronosphere API, use the [`ReadTraceBehaviorConfig`](/tooling/api-info/definition/operations/ReadTraceBehaviorConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Manage assigned behaviors You can manage the assigned behaviors for a dataset on two levels: * Assign a *main behavior* to define the primary behavior for a dataset. * Assign an *override behavior* to temporarily override the main behavior. You can assign only one main behavior and one override behavior to a dataset. Both the main and override layers can use any of the [trace behavior types](/control/shaping/sample-traces/behaviors#trace-behavior-types), which are baseline, allow, and deny. You can also [create custom behaviors](/control/shaping/sample-traces/behaviors#create-custom-behaviors) and assign them to the main or override layers on datasets. When assigning a behavior to the override layer, you can set the behavior to start immediately, or schedule it to start at a future time. When managing assigned behaviors, you can set the shaping order for overlapping trace datasets. The shaping order determines the priority order to apply behaviors when traces in one dataset overlap with traces in another dataset. For example, if a trace belongs to more than one dataset with an assigned behavior, Observability Platform uses the behavior assigned to the dataset that's first in the shaping order. The shaping order applies only when the selected behavior is active. Select from the following methods to assign behaviors to a dataset. If a behavior isn't currently assigned to a dataset, you must [assign behaviors](/control/shaping/sample-traces/datasets#assign-behaviors) to the dataset by selecting a dataset from the **Overview** tab of Trace Control Plane. To assign behaviors to a dataset: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. Click the **Behaviors** tab, and then click the name of the behavior you want to manage. 3. On the **Behavior Details** page, locate the dataset you want to manage behaviors for, click the three vertical dots icon, and select **Manage behaviors**. The selected behavior must be assigned to a dataset to manage it from the **Behaviors** tab of Trace Control Plane. 4. On the **Manage behaviors** page, in the **Main layer** pane, select a main behavior from the dropdown. 5. Optional: In the **Override layer** pane, select an override behavior and choose the override start and end time, and select a duration for how long the override remains active. 6. Select a shaping order for your main behavior. Shaping order is in decreasing priority, so a behavior in position one takes precedence over a behavior in position three. The shaping order section indicates the percentage of overlap between datasets so you can better understand the impact to traces in other datasets. 7. Click **Save** to save the behavior definition for your dataset. If you don't already have a YAML configuration file, use the `scaffold` Chronoctl parameter to generate a template for a specific resource type: ```shell theme={null} chronoctl trace-behavior-config scaffold ``` You can redirect the results (using the redirection operator `>`) to a file for editing. To assign a behavior with [Chronoctl](/tooling/chronoctl): 1. Run the following command to generate a sample behavior configuration you can use as a template: ```shell theme={null} chronoctl trace-behavior-config scaffold ``` In the template, `kind: TraceBehaviorConfig` defines an individual dataset. 2. With a completed definition, submit it with: ```shell theme={null} chronoctl trace-behavior-config create -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. See the [Chronoctl behavior example](/control/shaping/sample-traces/datasets#chronoctl-behavior-example) for a completed behavior definition. To complete this action with the Chronosphere API, use the [`UpdateTraceBehaviorConfig`](/tooling/api-info/definition/operations/UpdateTraceBehaviorConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Preview behaviors When viewing a dataset in Observability Platform, you can select a different behavior from the active behavior and preview its effects on the data volume of the selected dataset. This lets you temporarily preview a behavior to understand its impact before assigning it. On-call engineers can use this feature to show that a temporary override won't consume your organization's entire tracing license, which can let an administrator promote the preview behavior with confidence. When you preview a behavior, it runs for up to seven days or until you stop the preview. Preview behaviors don't shape datasets until you promote them. When promoted, preview behaviors inherit the same shaping order as your active behaviors. If no behaviors are active for the selected dataset for which you're previewing a behavior, the shaping order defaults to one, which is the highest priority. If you use datasets with tail sampling rules but have no active behaviors, any behaviors that you preview will default to shaping order `1` until you assign behaviors to your datasets. To preview behaviors: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. On the **Overview** tab, select the dataset you want to preview a behavior for. 3. On the selected dataset page, under **Preview behavior**, select the behavior you want to preview. 4. Click **Start** to start the preview. The preview starts running, and the volume license consumption charts display a dotted line indicating the effect of the preview if it's promoted. It can take up to two minutes before the dotted line displays on each of the charts. You might need to zoom in on the graphs to the current day or the last few minutes for the dotted line to display. 5. To stop the preview, click **Stop**. When previewing a behavior, an administrator can promote it. To promote the preview behavior, click **Promote** and then select one of these options: * **Main**: sets the preview behavior as the primary behavior for the dataset. * **Override**: sets the preview behavior as a temporary override for the main behavior. ## Customize your sampling strategy You can edit the facets of the baseline behavior to modify the tail sampling strategy you want to apply to your datasets. There are two main parts of the tail sampling strategy with different facets you can modify: drop less interesting, low-value traces and keep more interesting, high-value traces. * *Less interesting traces* might be ones that denote success. These traces indicate that your app is working as designed, but you likely don't need to keep most of them. * *More interesting traces* might be error traces, which indicate an issue that requires attention. Keeping more interesting traces helps when you're debugging issues and need to identify the source of the problem. There are several facets you can configure: * **Failed traces**: Traces containing root spans that have a status set to `Error`, as defined by the OpenTelemetry [span status](https://opentelemetry.io/docs/concepts/signals/traces/#span-status). * **Small traces**: Traces with very few spans that often indicate repeated messages about successful operations or incomplete instrumentation. * **Large traces**: Traces that are difficult to parse because of their large size, and which are rarely used in incident debugging due to a high ratio of noise to signal. These traces also consume a large amount of the persistence budget. * **Slow traces**: Traces that take a long time to complete, which can indicate issues in a related operation or service. * **Fast traces**: Traces that complete very quickly, which can be either repeated messages about successful operations or incomplete traces. Although each part of the baseline sampling strategy includes a set of facets by default, you can move facets from the low-value section to the high-value section, and the reverse. You can select the number of spans to keep, the duration for slow and fast traces, and the sampling rate for each facet. This flexibility lets you shape your sampling strategy as you learn more about your trace data and determine what information is valuable to your organization, and what isn't. You can disable some of the facets if you don't want to use them in your baseline strategy. For example, you don't want to keep most of your successful traces because you know they're successful, and you want to sample only a portion of your very small traces. In the **Default** section of the sampling strategy, you set the sample rate to 0.1%, and then set the sample rate for small traces to 10%. In the high-value section of the sampling strategy, you define the criteria for the traces you want to keep. The facets for this part of the strategy might include failed traces, large traces, and slow traces. When you assign your customized baseline behavior to a dataset, Observability Platform applies the following match criteria in order: 1. All traces in the dataset get matched against the behavior's low-value trace criteria. If a trace meets one or more of these criteria, the lowest possible sample rate applies. 2. All remaining traces in the dataset get matched against the high-value trace criteria. If a trace meets one or more of these criteria, the highest possible sampling rate applies. 3. If a trace doesn't match either of these criteria, the sample rate specified in the **Default** section applies. When viewing [span details](/investigate/querying/traces/features#span-details) in Trace Explorer, use the `tail_sampling_rate` tag on the root span of the trace to determine the tail sampling rate for the entire trace. The rate can be in a range of `0` to `1`, expressed as a decimal percentage. For example, a `tail_sampling_rate` of `0.01` equates to 1%, whereas a value of `0.25` equates to 25%. In the **Span details** section of Trace Explorer, use this tag in conjunction with the `behavior_dataset_slug` tag to understand which [dataset](/control/shaping/sample-traces/datasets) a trace inherits its sampling rate from. ### Edit the baseline behavior You can edit facets of the baseline behavior using the Observability Platform app only. To edit the baseline behavior: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. Click the **Behaviors** tab, and then click the **Baseline** behavior. 3. On the **Behavior details** panel, click **Edit**. 4. On the **Edit baseline behavior** panel, define your sampling methodology: * **Head sampling**: Specify the sample rate for head sampling to capture only a portion of possible traces from originating services. To specify a sample rate for head sampling in the baseline behavior, you must instrument [head sampling](/control/shaping/sample-traces/head-sampling) first. * **Tail sampling**: Specify the sample rate for each of the facets to drop low-value traces and keep high-value traces. Enable which facets to include in your tail sampling strategy, and specify a sampling rate for each facet. Specific sample rates depend on how your organization configures individual services. The following recommendations are based on best practices observed across many organizations: * **Large traces**: Set the number of spans to sample to `10,000`, and set a sample rate percentage of `25` or less. Some organizations might set the number of spans to sample closer to `3,000` and set the sample rate percentage to `5`. * **Slow traces**: Set the minimum duration threshold to `5` seconds and set a sample rate between `80` percent and `100` percent. * **Failed traces**: Set the sample rate between `80` percent and `100` percent to ensure that a repository of interesting error information is available for engineers investigating a problem. * **Small traces**: Set the number of spans to sample to `2`, and set a sample rate percentage of `50` or less. * **Fast traces**: Set the maximum duration threshold to `0.00001` seconds and set a sample rate percentage of `50` or less. To move facets between the low-value and high-value sections of the tail sampling strategy, click the left **** or right **** arrow for each facet you enable. * **Default**: Set the sample rate percentage to a mid-range value, such as `50`. If a trace doesn't match the low or high-value criteria, then the sample rate specified in the **Default** section applies. 5. Click **Save** to save the changes to your baseline behavior. ### Create custom behaviors Chronosphere provides specific [trace behavior types](#trace-behavior-types) your organization can use to define your sampling strategy for trace datasets. These behaviors let you determine which traces to drop and which to keep across assigned datasets. As you learn more about your trace data, you can create custom behaviors and set different tail sampling rates for individual datasets. Custom behaviors let you modify behaviors for each dataset, ensuring that you're spending the highest portion of your budget on the most critical and relevant data. To create a custom behavior, you duplicate the baseline behavior, define the tail sampling rules to apply, and set a default sampling rate for the behavior. You can then assign the custom behavior to a dataset. After creating a custom behavior, you can duplicate that behavior rather than duplicating the baseline behavior. To create a custom behavior: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. Click the **Behaviors** tab, and take one of the following actions to display the **Duplicate behavior** dialog: * In the **Chronosphere managed behaviors** section, in the row for the **Baseline** behavior, click ** Duplicate**. * Click the **Baseline** behavior, and then on the **Baseline** page, click ** Duplicate**. 3. On the **Duplicate behavior** panel, enter a name and description for your custom behavior. Observability Platform generates a slug based on the name you enter. You can modify the slug to any alphanumeric combination, but special characters aren't supported. 4. Modify the tail sampling attributes to reflect the traces you want to drop or keep. See [edit the baseline behavior](#edit-the-baseline-behavior) for recommendations based on best practices observed across many organizations. 5. Set the default sample rate percentage. If a trace doesn't match the low or high-value tail sampling attributes you defined, then the default sample rate applies. 6. Click **Save**. After creating a custom behavior, you can [assign the behavior](/control/shaping/sample-traces/datasets#assign-behaviors) to a dataset. ### Modify custom behaviors After [creating a custom behavior](#create-custom-behaviors), you can modify its attributes. For example, you might want to modify the tail sampling strategy of a behavior as you learn more about the dataset the behavior is assigned to. To modify a custom behavior: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. Click the **Behaviors** tab, and take one of the following actions to display the **Edit behavior** dialog: * In the **Custom behaviors** section, in the row for the behavior you want to modify, click ** Edit**. * Click the behavior you want to modify, and then on the selected behavior page, click ** Edit**. 3. Modify the tail sampling attributes to reflect the traces you want to drop or keep. See [edit the baseline behavior](#edit-the-baseline-behavior) for recommendations based on best practices observed across many organizations. 4. Click **Save**. ### Delete custom behaviors To delete a behavior that's assigned to one or more datasets, you must first assign all active and scheduled datasets to a different behavior. Observability Platform won't let you delete a behavior that's assigned to a dataset. To delete a custom behavior: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. Click the **Behaviors** tab, and take one of the following actions to display the **Delete behavior** dialog: * In the **Custom behaviors** section, in the row for the behavior you want to delete, click ** Delete**. * Click the behavior you want to delete, and then on the selected behavior page, click ** Delete**. 3. On the **Delete behavior** confirmation dialog, click **Delete**. Observability Platform deletes the custom behavior. # Trace datasets Source: https://docs.chronosphere.io/control/shaping/sample-traces/datasets Use datasets in Chronosphere Observability Platform to map sets of traces to named groups relevant to your organization. Understanding your tracing license consumption helps identify where you're spending the most money on your tracing data. Trace *datasets* are a control mechanism that let you map sets of traces to named groups relevant to your organization, and then track processed and persisted bytes for those groups over time. For example, you might create a Shopper dataset based on data like services, operations, customer IDs, and tags that relate to your shopping app. Viewing that dataset provides a snapshot of trace data volume associated with the entire business unit related to your shopping app. Chronosphere recommends creating one dataset per team or per environment. Understanding data consumption for individual business units can highlight which [sampling rules](/control/shaping/sample-traces) to adjust so you can better control your trace data license consumption and remain within defined data limits. Datasets are part of the Trace Control Plane, which also includes trace behaviors and head and tail sampling rules. You need administrative access to use the Trace Control Plane. To access trace datasets, in the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. When [searching traces](/investigate/querying/traces#search-optimizations) in Trace Explorer, use the **Custom tags** field in the **Query builder** in Trace Explorer to search for the `behavior_dataset_slug` tag. This tag identifies traces that were sampled by a particular dataset's behavior during processing. A trace cam match many datasets, but will only be sampled by the behaviors on the first dataset it matched. used to determine the active behavior for sampling purposes. ## View datasets Select from the following methods to view and filter available trace datasets. To view trace datasets: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. The **Overview** tab displays your total license consumption for the selected period, which defaults to the current month to date. This view includes graphs that display the daily volume breakdown and the cumulative breakdown over the current week. 2. Take any of the following actions to change the displayed data: * On either of the **Processed** or **Persisted** graphs, click the more icon and select **Open in Metrics Explorer** to visualize the underlying query. * Toggle **Show unique volume** to display only the volume of data that doesn't overlap with another dataset. * Toggle **Show dropped volume** to display only the volume of data that's being dropped. 3. Use the search box to search for a specific dataset. The row for each dataset displays, the total data volume, the percent of data overlap, and any active behaviors. 4. In the datasets table, select one or more datasets to update the graphs. You can click and drag a section of either graph to zoom in on the selected time period. 5. To view an individual dataset, click the name of the dataset you want to view from the list. The individual dataset page includes a definition of the underlying Trace Explorer query and the services at the root of all traces in the dataset. To view the underlying queries, in either the **Definition** or **Root services**, click **Search in Trace Explorer**. To use [Chronoctl](/tooling/chronoctl) to return all trace datasets, use the `chronoctl datasets list` command: ```shell theme={null} chronoctl datasets list ``` To filter for a specific trace dataset, add the `slugs` argument to the command: ```shell theme={null} chronoctl datasets list --slugs SLUG ``` Replace *`SLUG`* with the slug of the dataset you want to display. [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) tool to view the dataset's Chronoctl YAML representation. [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) in Observability Platform to view a dataset's Terraform representation. To complete this action with the Chronosphere API, use the [`ListDatasets`](/tooling/api-info/definition/operations/ListDatasets) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create datasets To create a dataset, define and test your trace query, and then map that query to the resource you want to create. After creating datasets, you can [assign trace behaviors](#assign-behaviors) for your datasets. Behaviors let you set sampling rates and the shaping order, which determines the order of priority behaviors to apply when there are overlapping traces with other datasets. Use one of the following methods to create a dataset. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. Click **Create dataset**. 3. Enter a display name for your dataset, which is used to generate a default slug. If you want the slug to be a different value, edit the **Slug** field directly. 4. Enter comments about the dataset, such as the business unit this dataset tracks trace data for. 5. Define dataset match criteria to outline the query that matches traces you want included in the dataset. You can add one or more span filters to additionally refine the trace results. See [Search and filter trace data](/investigate/querying/traces) for information about how to define an effective search for trace data. 6. Click **View statistics** to open Trace Explorer in a new tab with your defined query. Review the results to ensure your query returns the trace data you expect. 7. In the **Create dataset** pane, click **Save** to create your dataset. Observability Platform creates your dataset and displays its definition. Next, [assign trace behaviors](#assign-behaviors) for your dataset to set sampling rates. If you don't already have a YAML configuration file, use the `scaffold` Chronoctl parameter to generate a template for a specific resource type: ```shell theme={null} chronoctl datasets scaffold ``` You can redirect the results (using the redirection operator `>`) to a file for editing. To create a dataset with [Chronoctl](/tooling/chronoctl): 1. Define a query in [Trace Explorer](/investigate/querying/traces) that represents the data you want included in the dataset. For example, the following query returns all traces where at least one span includes a service called `payment-svc`, an operation that starts with `checkout`, and a tag named `env=prod`: ```text theme={null} service="payment-svc" operation=~"^payment*." tag:env=prod*" ``` 2. Create a YAML definition to map the query to a dataset that represents the business unit you want to track trace data for. Use the following command to generate a sample dataset configuration you can use as a template: ```shell theme={null} chronoctl datasets scaffold ``` In the template, `kind: Dataset` defines an individual dataset. 3. With a completed definition, submit it with: ```shell theme={null} chronoctl datasets create -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. See the [Chronoctl dataset example](#chronoctl-dataset-example) for a completed dataset definition. After creating your dataset, [assign trace behaviors](#assign-behaviors) for your dataset to set sampling rates and the shaping order, which determines the order of priority behaviors to apply when there are overlapping traces with other datasets. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. To create a dataset with [Terraform](/tooling/infrastructure/terraform): 1. Define a query in [Trace Explorer](/investigate/querying/traces) that represents the data you want included in the dataset. For example, the following query returns all traces where at least one span includes a service called `payment-svc`, an operation that starts with `checkout`, and a tag named `env=prod`: ```text theme={null} service="payment-svc" operation=~"^payment*." tag:env=prod*" ``` 2. Create a Terraform resource to map the query to a dataset that represents the business unit you want to track trace data for. 3. Add the definition by using the `chronosphere_dataset` type, followed by a name in a resource declaration. 4. Run this command to apply the changes: ```shell theme={null} terraform apply ``` See the [Terraform dataset example](#terraform-dataset-example) for a completed dataset resource. To complete this action with the Chronosphere API, use the [`CreateDataset`](/tooling/api-info/definition/operations/CreateDataset) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Identify incomplete traces You can create a dataset specifically for identifying incomplete traces, which are traces with spans that reference other spans outside of the selected trace. Incomplete traces can occur if a service is misconfigured and isn't exporting spans correctly. Chronosphere recommends creating at least one dataset with the Chronosphere-supplied `parent_missing=true` key-value pair to help identify and track changes in incomplete trace volume or trace instrumentation over time. As you add more trace instrumentation, fewer traces meet this criteria, which drives down the volume of traces in this dataset. You can also apply behaviors to this dataset to decrease the persisted volume of incomplete traces. Use one of the following examples to create a dataset for identifying incomplete traces. ```yaml theme={null} name: Partial Traces slug: partial-traces description: Track data volume for incomplete traces. configuration: type: TRACES trace_dataset: match_criteria: span: - match_type: INCLUDE tags: - key: parent_missing value: match: EXACT value: true ``` ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_dataset" "incomplete_traces" { name = "Incomplete traces" description = "Track data volume for incomplete traces." configuration { type = "TRACES" trace_dataset { match_criteria { span { match_type = "INCLUDE" tag { key = "parent_missing" value { value = "true" match = "EXACT" } } } } } } } ``` ### Chronoctl dataset example The following YAML definition consists of one dataset named `Traces payment service US prod`. This dataset includes any spans that include the `payment` service, the `payment_store` operation, and have a tag where `deployment.environment=production`. If you want to specify criteria at the trace level rather than the span level, define `trace` instead of `span` in your YAML definition. ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Dataset spec: # Required name of the dataset. Can be modified after the dataset is created. name: Traces payment service US prod # Unique identifier of the dataset. If not provided, a slug is generated based # on the name field. Can't be modified after the dataset is created. slug: traces-payment-service-us-prod # Optional description for the dataset. description: Traces for payment service in US production environment # Defining characteristics of the dataset. configuration: # Dataset type, which must be TRACES. type: TRACES trace_dataset: # Trace criteria to match for the dataset. match_criteria: # Object that represents the span conditions to match on. All conditions must # be true in a single span for the span to be considered a match. span: # Determines whether in INCLUDE or EXCLUDE all traces that contain at least # one span matching the filter. - match_type: INCLUDE # The service to match on in candidate spans. service: # Operator to compare in_values with. Can be one of EXACT, REGEX, # EXACT_NEGATION, REGEX_NEGATION, IN, NOT_IN. match: IN # Values the filter tests against when using IN or NOT_IN match type. in_values: - payment # The operation to match on in candidate spans. operation: match: REGEX # The value the filter compares to the target trace or span field. value: /payment_store/.* # The tag to match on in candidate spans. tags: # The key of the span tag to match on in the filter. - key: deployment.environment value: match: EXACT value: production ``` ### Terraform dataset example The following Terraform resource creates a dataset that Terraform refers to by `prod_payment_us`, and with a human-readable name of `Traces payment service US prod`. This dataset includes any spans that include the `payment` service, where the parent service matches either `us-east` or `us-west`, the parent operation begins with `/payment`, and a tag where `environment` includes `prod`. If you want to specify criteria at the trace level rather than the span level, define `trace` instead of `span` in your YAML definition. ```terraform expandable Terraform example icon="square-terminal" theme={null} resource "chronosphere_dataset" "prod_payment_us" { # Required name of the dataset. Can be modified after the dataset is created. name = "Traces payment service US prod" # Optional description for the dataset. description = "Traces passing through the payment service in US production" # Defining characteristics of the dataset. configuration { # Dataset type, which must be TRACES. type = "TRACES" trace_dataset { # Trace criteria to match for the dataset. match_criteria { # Object that represents the span conditions to match on. All conditions must # be true in a single span for the span to be considered a match. span { # Matches traces based on the entire duration of the trace. duration { max_secs = 99 min_secs = 1 } # Matches traces based on the top-level error status. error { value = true } # Determines whether in INCLUDE or EXCLUDE all traces that contain at least # one span matching the filter. match_type = "INCLUDE" # Matches the operation of the candidate span's parent span if it's not a # root span. parent_operation { value = "payments/.*" match = "REGEX" } # Matches the service of the candidate span's parent span if it's not a # root span. parent_service { value = "us-[east|west]" match = "REGEX" } # The service to match on in candidate spans. service { match = "IN" in_values = ["payment"] } # Defines the number of spans that must match the criteria defined by # filter. Defaults to least one span. span_count { max = 2 min = 1 } # The tag to match on in candidate spans. tag { key = "environment" value { value = "prod.*" match = "REGEX" } } tag { key = "client_build" value { match = "NOT_IN" in_values = ["debug", "beta"] } } } } } } } ``` ## Assign behaviors When viewing an individual dataset, you can assign a [behavior](/control/shaping/sample-traces/behaviors) to the dataset to set sampling rates on two levels: * Assign a *main behavior* to define the primary behavior for a dataset. * Assign an *override behavior* to temporarily override the main behavior. You can assign only one main behavior and one override behavior to a dataset. Both the main and override layers can use any of the [trace behavior types](/control/shaping/sample-traces/behaviors#trace-behavior-types), which are baseline, allow, and deny. You can also [create custom behaviors](/control/shaping/sample-traces/behaviors#create-custom-behaviors) and assign them to the main or override layers on datasets. When assigning a behavior to the override layer, you can set the behavior to start immediately, or schedule it to start at a future time. When managing assigned behaviors, you can set the shaping order for overlapping trace datasets. The shaping order determines the priority order to apply behaviors when traces in one dataset overlap with traces in another dataset. For example, if a trace belongs to more than one dataset with an assigned behavior, Observability Platform uses the behavior assigned to the dataset that's first in the shaping order. The shaping order applies only when the selected behavior is active. Assigning a behavior to a dataset is different than [editing the baseline behavior](/control/shaping/sample-traces/behaviors#customize-your-sampling-strategy), where you can modify the facets based on the sampling strategy you want to use. Select from the following methods to assign behaviors to a dataset. To assign behaviors to a dataset: You can also [manage assigned behaviors](/control/shaping/sample-traces/behaviors#manage-assigned-behaviors) from the **Behaviors** tab of Trace Control Plane. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. From the list of datasets, click the dataset you want to manage behaviors for. 3. In the selected dataset page, in the **Behavior** pane, click **Manage**. If you already have a behavior assigned to a dataset, you can [run a preview](/control/shaping/sample-traces/behaviors#preview-behaviors) of another dataset to preview its affects based on a dataset's volume. This capability lets you temporarily preview a behavior to understand its impact before assigning it. 4. In the **Main layer** pane, select a main behavior from the dropdown. 5. Optional: In the **Override layer** pane, select an override behavior and choose when the override should start and end, and select a duration for how long the override remains active. 6. Select a shaping order for your main behavior. Shaping order is in decreasing priority, so a behavior in position one takes precedence over a behavior in position three. 7. Click **Save** to save the behavior definition for your dataset. If you don't already have a YAML configuration file, use the `scaffold` Chronoctl parameter to generate a template for a specific resource type: ```shell theme={null} chronoctl trace-behavior-config scaffold ``` You can redirect the results (using the redirection operator `>`) to a file for editing. To assign a behavior with [Chronoctl](/tooling/chronoctl): 1. Run the following command to generate a sample behavior configuration you can use as a template: ```shell theme={null} chronoctl trace-behavior-config scaffold ``` In the template, `kind: TraceBehaviorConfig` defines an individual dataset. 2. With a completed definition, submit it with: ```shell theme={null} chronoctl trace-behavior-config create -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. See the [Chronoctl behavior example](#chronoctl-behavior-example) for a completed behavior definition. To complete this action with the Chronosphere API, use the [`UpdateTraceBehaviorConfig`](/tooling/api-info/definition/operations/UpdateTraceBehaviorConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Chronoctl behavior example The following YAML definition consists of one behavior named `Traces payment service US prod`. This dataset includes any spans that include the `payment` service, the `payment_store` operation, and have a tag where `deployment.environment=production`. ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: TraceBehaviorConfig spec: # List of assignments for the main behavior. The referenced datasets are datasets # to enroll in behaviors. The referenced behaviors are the active behaviors # for the dataset when there is no override in place. # * Only one main behavior can be assigned to a dataset. # * Only one referenced 'TraceBehavior' with 'type' field set to 'TYPE_BASELINE' can # be set, which must match the slug referenced by 'baseline_behavior_slug'. main_behavior_assignments: - created_at: "2024-08-24T14:15:22Z" updated_at: "2024-08-24T13:22:21Z" # The slug reference of a TraceDataset dataset_slug: "shopper-dataset" # The slug reference of a TraceBehavior behavior_slug: "baseline" # The author or creator of the entry. created_by: "someone@example.com" # A description of the entry. description: "Description of the behavior" # List of assignments for the override behavior. OverrideBehaviorAssignments are used to # specify the active behavior for a dataset over a specific time range. # * Only one override behavior can be assigned to a dataset. # * Only one referenced 'TraceBehavior' with 'type' field set to 'TYPE_BASELINE' can # be set, which must match the slug referenced by 'baseline_behavior_slug', and any # baseline behavior referenced in 'main_behavior_assignments'. override_behavior_assignments: - created_at: "2024-08-24T14:15:22Z" updated_at: "2024-08-24T13:22:21Z" # The slug reference of a TraceDataset dataset_slug: "shopper-dataset" # The slug reference of a TraceBehavior behavior_slug: "keep-all" # The starting time of the override. start_time: "2024-08-26T14:15:22Z" # The ending time of the override. end_time: "2024-08-26T15:15:22Z" # The author or creator of the entry. created_by: "someone@example.com" # A description of the entry. description: "Allow all traces for one hour" # List of dataset priorities. This list specifies the order in which datasets # are considered when determining the behavior to follow for a trace. Dataset # priorities are used to break ties when a trace matches more than one dataset # with an active behavior. # * Each entry in this list must refer to the slug of an existing dataset. # * The order of the list is the order in which the datasets are considered. # * The list must contain all datasets referenced in either main_behavior_assignments # and override_behavior_assignments. # * The list may contain datasets that are not referenced in either of the # previous references. dataset_priorities: - "baseline" - "keep-all" # The baseline behavior to use for behavior assignments and base head sampling rates. # Must reference a TraceBehavior entity with type: TYPE_BASELINE. baseline_behavior_slug: "baseline" ``` ## Edit datasets When creating or editing a dataset, you can [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to view code representations of a dataset for [Terraform](/tooling/infrastructure/terraform), [Chronoctl](/tooling/chronoctl), and the [Chronosphere API](/tooling/api-info). The displayed code also responds to changes you make in the **Visual Editor** tab. Entities modified by Terraform and Chronoctl are viewable in Observability Platform, but can't be modified. Select from the following methods to edit trace datasets. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. From the list of datasets, click the dataset you want to edit. 3. On the selected dataset page, click **Edit dataset**. 4. Make changes to your dataset, and then click **Save**. Observability Platform saves changes to your dataset. To edit datasets using [Chronoctl](/tooling/chronoctl): 1. [View the datasets Chronoctl YAML](#view-datasets). 2. Modify its properties and apply the changes with the same process as [Creating datasets](#create-datasets). Chronoctl updates the dataset if it has the same slug. You can also use the following process if you already have a definition file: 1. Update the datasets definition file. 2. Run the following command to submit the changes: ```shell theme={null} chronoctl datasets update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. You can also [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to view the dataset's Chronoctl YAML representation. To edit datasets using [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file that updates the resource's existing properties. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` You can also [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to view the dataset's Terraform representation. To complete this action with the Chronosphere API, use the [`UpdateDataset`](/tooling/api-info/definition/operations/UpdateDataset) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete datasets Select from the following methods to delete trace datasets. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). Complete the following steps before you delete a dataset: 1. [Remove any assigned behaviors](#assign-behaviors). 2. Stop any active [preview behaviors](/control/shaping/sample-traces/behaviors#preview-behaviors). After removing any assigned behaviors and stopping active preview behaviors, delete the dataset: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. From the list of datasets, click the dataset you want to delete. 3. On the selected dataset page, click **Delete dataset**. 4. In the confirmation dialog, click **Delete** to delete the dataset. To delete a dataset with [Chronoctl](/tooling/chronoctl), use the `chronoctl datasets delete` command: ```shell theme={null} chronoctl datasets delete SLUG ``` Replace *`SLUG`* with the slug of the dataset you want to delete. For example, to delete a dataset with the slug `infra-example-dataset`: ```shell theme={null} chronoctl datasets delete infra-example-dataset ``` To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteDataset`](/tooling/api-info/definition/operations/DeleteDataset) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # Head sampling Source: https://docs.chronosphere.io/control/shaping/sample-traces/head-sampling Learn how to configure head sampling to capture a portion of your traces from originating services. You can configure dynamic, remotely configurable [*head sampling*](https://opentelemetry.io/docs/concepts/sampling/#head-sampling) to capture only a portion of possible traces from originating services. Use head sampling rules to limit possible traces to a fraction of all possible traces. You define sampling strategies to target specific kinds of traces, focus on specific services, or concentrate on a combination of service and operation. Head sampling applies only to root services, which are services that appear on the root span of a trace. Identifying root services can help identify where the most requests to a service originate, and can inform which service to increase sampling for to generate additional trace data. To instrument head sampling, Chronosphere Observability Platform supports the [OpenTelemetry JaegerRemoteSampler](https://opentelemetry.io/docs/specs/otel/trace/sdk/#jaegerremotesampler) head sampling standard. To use head sampling, you must first [configure the OpenTelemetry Collector](#configure-the-opentelemetry-collector). ## View head sampling rules View head sampling rules in Observability Platform to understand how each rule impacts your tracing data, such as the sampling rate, rule criteria, and the impact the rule is having on your trace data volume. To return a list of defined head sampling rules without additional information, use Chronoctl. In Observability Platform, each head sampling rule indicates which service or combination of service and operation it impacts. A single rule can impact multiple operations at different sampling rates for a specific service. You need administrative access to complete this task. To view head sampling rules: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. Click the **Head sampling** tab. * The **Status** column indicates whether each rule is active or inactive. An active status means that a client fetched the head sampling rule within the last 30 minutes. * The **Traces Kept** column indicates the percentage at which the current rule is sampling your traces for the indicated service. Hold the pointer over the bar in this column to view a description for each rule. 3. Expand each of the configured rules to view the rule criteria and impact on your tracing data. You can also view which operations the rule impacts for the selected service. 4. If a rule impacts multiple operations, use the dropdown menu to select which operation to display impact data for. 5. Use the search box to locate rules impacting a specific service. To display a list of all available head sampling rules with [Chronoctl](/tooling/chronoctl), use this command: ```shell theme={null} chronoctl trace-jaeger-remote-sampling-strategies list ``` To complete this action with the Chronosphere API, use the [`ListTraceJaegerRemoteSamplingStrategies`](/tooling/api-info/definition/operations/ListTraceJaegerRemoteSamplingStrategies) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Configure head sampling To instrument head sampling, Observability Platform supports the [OpenTelemetry JaegerRemoteSampler](https://opentelemetry.io/docs/specs/otel/trace/sdk/#jaegerremotesampler) head sampling standard. Observability Platform implements the [JaegerRemoteSampler configuration API](https://www.jaegertracing.io/docs/1.41/apis/#remote-sampling-configuration-stable), and serves sampling strategies based on the Terraform resources you define. Observability Platform is compatible with the [Jaeger Remote Sampling extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/jaegerremotesampling) to enable the OpenTelemetry Collector to act as a proxy between OpenTelemetry agents and the Observability Platform backend. To configure head sampling: 1. [Configure the OpenTelemetry Collector](#configure-the-opentelemetry-collector). 2. [Configure your app as an OpenTelemetry agent](#configure-your-app-as-an-opentelemetry-agent). 3. [Create head sampling rules](#create-head-sampling-rules). ### Configure the OpenTelemetry Collector > Requires OpenTelemetry Collector version 0.83 or later. Before configuring OpenTelemetry agents, you must configure the OpenTelemetry Collector to export tracing data to Chronosphere. 1. [Configure the OpenTelemetry Collector](/ingest/metrics-traces/otel/otel-ingest) to export tracing data to Observability Platform. 2. In the OpenTelemetry Collector `config.yaml` file, apply the following settings to modify the `extensions` YAML collection. Specify an `endpoint` that points to your Observability Platform tenant, and include the Chronosphere API token you created for the OpenTelemetry Collector. Chronosphere recommends specifying a `reload_interval` to control a local cache of your sampling strategy for the `remote` source (which is Observability Platform). This interval reduces the number of backend API calls the OpenTelemetry Collector makes for your instrumented agents, which helps reduce egress costs. ```yaml /TENANT/ /API_TOKEN/ theme={null} extensions: zpages: endpoint: 0.0.0.0:55679 health_check: {} pprof: {} jaegerremotesampling: source: reload_interval: 30s remote: endpoint: TENANT compression: gzip headers: API-Token: ${env:API_TOKEN} ``` * * ### Configure your app as an OpenTelemetry agent Configure your app as an [OpenTelemetry agent](https://opentelemetry.io/docs/collector/deployment/agent/) to act as a proxy for the OpenTelemetry Collector, which sends your tracing data to Observability Platform. Instrument your app with the OpenTelemetry SDK using the [OpenTelemetry protocol (OTLP)](https://opentelemetry.io/docs/specs/otel/protocol/). Your app sends spans to the OpenTelemetry Collector, which pulls head sampling rules from Observability Platform. As you update head sampling rules in Observability Platform, the OpenTelemetry Collector pulls the changes and transmits the updated rules to your app. ```mermaid actions={false} theme={null} --- title: Span flow with head sampling rules --- flowchart LR A["Agent (your app)" instrumented with OTLP] A -- spans --> B A -. sampling rules .-> B B[OpenTelemetry Collector] --> C[Observability Platform] D[Head sampling rules] D -.-> C B -.-> D ``` The implementation depends on the programming language of your app. For example, the following code from the Go implementation of OpenTelemetry, [OpenTelemetry-Go](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/samplers/jaegerremote/README.md), defines how to implement the JaegerRemoteSampler in Go: ```go /SERVICE_NAME/ /HOST_NAME/ theme={null} jaegerRemoteSampler := jaegerremote.New( "SERVICE_NAME", jaegerremote.WithSamplingServerURL("http://HOST_NAME:5778/sampling"), jaegerremote.WithSamplingRefreshInterval(10*time.Second), jaegerremote.WithInitialSampler(trace.TraceIDRatioBased(0.5)), ) tp := trace.NewTracerProvider( trace.WithSampler(jaegerRemoteSampler), // ... ) otel.SetTracerProvider(tp) ``` * *`SERVICE_NAME`*: Name of the service you're sampling, which can map to a microservice in your architecture. * *`HOST_NAME`*: Host name where your OpenTelemetry Collector is running. When configuring the Jaeger remote sampler, you must include all of these properties: * **Endpoint**: Endpoint where your OpenTelemetry Collector is running, which includes the Jaeger Remote Sampling extension that points to Observability Platform. * **Polling interval**: Interval at which your OpenTelemetry agents sync strategies from your OpenTelemetry Collector. * **Initial sampler**: Policy to implement from your service until the configured endpoint pulls a strategy. ## Create head sampling rules After configuring the OpenTelemetry Collector to export traces, configuring head sampling, and configuring your OpenTelemetry agents, manage your head sampling strategy using the [Chronosphere Terraform provider](/tooling/infrastructure/terraform) or [Chronoctl](/tooling/chronoctl). This ability means you can push strategy changes to all of your OpenTelemetry agents without modifying the JaegerRemoteSampler strategy directly. If you don't define a sampling strategy for a service, Observability Platform applies the default sampling rate of `0.001` (0.1%) to the service. The `applied_strategy` defines your sampling strategy, and can be one of the following values: * `probabilistic_strategy`: Defines a probabilistic strategy, which samples traces from the identified service based on the `sampling_rate`. This value determines the probability of sampling any trace, and must be in the range of `0` to `1`. * `rate_limiting_strategy`: Defines a rate-limited strategy that sets the maximum number of traces to sample per second. * `per_operation_strategies`: Defines a probabilistic strategy that sets a default sampling rate, plus an upper and lower bound. When creating your head sampling strategy with Chronoctl, consider the following behaviors of the `slug` property: * If you don't provide a `slug`, Chronoctl generates one based on the `name` field. * After creating your head sampling strategy, you can modify the `name` but can't modify the `slug`. * The `service_name` and `slug` must match. To define your head sampling strategy with [Chronoctl](/tooling/chronoctl): 1. Create a YAML file and define your head sampling strategy. Use the `chronoctl trace-jaeger-remote-sampling-strategies scaffold` command to generate an example resource for a head sampling rule. The following head sampling strategy defines a probabilistic strategy that sets a default sampling rate for the `inventory-operation`, plus an upper and lower bound: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: TraceJaegerRemoteSamplingStrategy spec: slug: ordering-service-sampling name: Ordering service head sampling strategy service_name: ordering-service-sampling applied_strategy: per_operation_strategies: default_sampling_rate: 0.01 default_lower_bound_traces_per_second: 1 default_upper_bound_traces_per_second: 1000 per_operation_strategies: - operation: inventory-operation probabilistic_sampling_strategy: sampling_rate: 0.1 ``` 2. Apply your head sampling strategy and send it to Observability Platform: ```shell /FILE_NAME/ theme={null} chronoctl apply -f FILE_NAME.yml ``` *`FILE_NAME`* is the name of your head sampling YAML file. Create a head sampling rule with Terraform by using the `chronosphere_trace_jaeger_remote_sampling_strategy` type in a `resource` declaration. When managing your head sampling strategy with Terraform, Chronosphere recommends defining one `resource` per microservice in your Terraform file. If you use the Chronosphere-managed JaegerRemoteSampler configuration API, you can create and change the sampling strategies of your services by running `terraform apply`, without having to redeploy your services or OpenTelemetry Collector. To define your head sampling strategy: 1. In your Terraform file, create your head sampling strategy. You must include a `name`,`service_name`, and `applied_strategy`. For example: ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_trace_jaeger_remote_sampling_strategy" "billing-service" { name = "billing-service Jaeger Remote Sampling strategy" service_name = "billing-service" applied_strategy { probabilistic_strategy { sampling_rate = 0.01 } } } ``` Refer to [this example](#terraform-example) for more information. 2. Apply the changes: ```terraform icon="square-terminal" theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`CreateTraceJaegerRemoteSamplingStrategy`](/tooling/api-info/definition/operations/CreateTraceJaegerRemoteSamplingStrategy) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Terraform example The following example provides a `resource` definition for three services and defines a distinct sampling strategy for each: * The first resource samples the `billing-service` service at a rate of `0.01`, which is 1% of traces. * The second resource samples at most two traces per second, per instance, from the `cart-service` service. For example, if the `cart-service` service consists of 17 pods, the expected sample of traces per second is somewhere between 0 and 34. * The third resource samples traces from the `ordering-service` service at a rate of `0.01`, or 1% of traces. If volumes are low, Observability Platform samples traces at least once per second. If volumes are high, Observability Platform stops sampling traces after reaching 1,000 traces per second. ```terraform expandable Terraform example icon="square-terminal" theme={null} resource "chronosphere_trace_jaeger_remote_sampling_strategy" "billing-service" { name = "billing-service Jaeger Remote Sampling strategy" service_name = "billing-service" applied_strategy { probabilistic_strategy { sampling_rate = 0.01 } } } resource "chronosphere_trace_jaeger_remote_sampling_strategy" "cart-service" { name = "cart-service Jaeger Remote Sampling strategy" service_name = "cart-service" applied_strategy { rate_limiting_strategy { max_traces_per_second = 2 } } } resource "chronosphere_trace_jaeger_remote_sampling_strategy" "ordering-service" { name = "ordering-service Jaeger Remote Sampling strategy" service_name = "ordering-service" applied_strategy { per_operation_strategies { default_sampling_rate = 0.01 default_lower_bound_traces_per_second = 1 default_upper_bound_traces_per_second = 1000 per_operation_strategies { operation = "notification-operation" probabilistic_strategy { sampling_rate = 0.0 } } per_operation_strategies { operation = "inventory-operation" probabilistic_strategy { sampling_rate = 0.1 } } per_operation_strategies { operation = "payment-operation" probabilistic_strategy { sampling_rate = 1.0 } } } } } ``` ## Edit head sampling rules Select from the following methods to edit head sampling rules. To edit head sampling rules with [Chronoctl](/tooling/chronoctl): 1. [View the head sampling rules Chronoctl YAML](#view-head-sampling-rules). 2. Modify its properties and apply the changes with the same process as [creating head sampling rules](#create-head-sampling-rules). Chronoctl updates head sampling rules if it has the same slug. You can also use the following process if you already have a definition file: 1. Update the head sampling rules definition file. 2. Run the following command to submit the changes: ```shell /FILE_NAME/ theme={null} chronoctl trace-jaeger-remote-sampling-strategies update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. To edit head sampling rules using [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file that updates the resource's existing properties. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`UpdateTraceJaegerRemoteSamplingStrategy`](/tooling/api-info/definition/operations/UpdateTraceJaegerRemoteSamplingStrategy) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete head sampling rules Select from the following methods to delete head sampling rules. To delete a head sampling rule with [Chronoctl](/tooling/chronoctl), use this command: ```shell /SLUG/ theme={null} chronoctl trace-jaeger-remote-sampling-strategies delete SLUG ``` Replace *`SLUG`* with the slug of the head sampling rule you want to delete. For example, to delete a head sampling rule with the slug `head-sampling-prod`: ```shell theme={null} chronoctl trace-jaeger-remote-sampling-strategies delete head-sampling-prod ``` To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteTraceJaegerRemoteSamplingStrategy`](/tooling/api-info/definition/operations/DeleteTraceJaegerRemoteSamplingStrategy) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # Tail sampling Source: https://docs.chronosphere.io/control/shaping/sample-traces/tail-sampling Learn about how to configure tail sampling to configure filters on trace data before it's stored. You can configure *tail sampling* to apply a set of fine-grained rules after any [head sampling](/control/shaping/sample-traces/head-sampling) decisions. While head sampling allows probabilistic sampling at the start of a trace, tail sampling focuses on the result of a trace. You can implement rules to look at the downstream effects of an operation, and evaluate traces based on outcomes, such as whether an error occurred or if a trace contributed to higher latency than normal. After evaluating these rules, you can keep a higher percentage of influential traces while downsampling baseline traces. Use tail sampling to configure specific filters on trace data before it's stored, and then sample your data based on those rules. For example, you might create a rule to keep all error traces, or continue downsampling all successful traces. These types of rules help to reduce costs and limit the amount of information you need to triage when debugging issues. For more information about tail sampling, see the [Tail Sampling](https://opentelemetry.io/docs/concepts/sampling/#tail-sampling) section of the OpenTelemetry Sampling documentation page. ## View tail sampling rules In Chronosphere Observability Platform, you can view the tail sampling rules you configured in Terraform to understand the impact of each rule on your tracing data. These impacts can include the sampling rate, the rule's criteria, and the impact of a rule on your incoming traces. Observability Platform evaluates each trace against each rule's trace filter, in order of precedence, until a rule matches. If a rule matches, Observability Platform applies the matched rule's sampling rate to the trace. If a trace doesn't match any rules, Observability Platform applies the default sample rate to the trace. If a default sampling rate isn't specified, Observability Platform keeps all traces. You need administrative access to complete this task. To view tail sampling rules: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Control Plane**. 2. Select the **Tail sampling** tab. The sample rate for each rule displays in the **Traces Kept** column, in addition to the **Created** and **Updated** dates. Hold the pointer over the bar in the **Traces Kept** column to view a description for each rule. 3. Expand each of the configured rules to view the rule criteria and impact on your tracing data. 4. Use the search box to locate rules impacting a specific service or operation. To display a list of all available tail sampling rules with [Chronoctl](/tooling/chronoctl), use this command: ```shell theme={null} chronoctl trace-tail-sampling-rules read ``` To complete this action with the Chronosphere API, use the [`ReadTraceTailSamplingRules`](/tooling/api-info/definition/operations/ReadTraceTailSamplingRules) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create tail sampling rules You can create tail sampling rules using the [Chronosphere Terraform provider](/tooling/infrastructure/terraform) or [Chronoctl](/tooling/chronoctl). You create one set of tail sampling rules as an ordered list. Rules are evaluated in match order. In your rule definition file, put broader rules at the top, such as a rule that drops any traces with [health check data](#drop-all-health-check-traces). For each rule, complete the following tasks: * Assign a human-readable `name` to identify the tail sampling rule in Observability Platform. * Assign a `system_name`, which provides a unique label name for the metric group that traces affected by the rule. * Define a specific filter such as `"error=true"`. * Specify a sampling rate. The defaults sampling rate is `1`, which means that Observability Platform stores all traces that don't match any sampling rules. Sampling rates must be a number between `0` and `1`, where a rate of `0` drops all traces, and a rate of `1` keeps all traces matching the defined filter. A sampling rate of `.5` drops half of all traces matching the filter, and keeps the other half. For a complete list of supported fields for tail sampling rules, see the [CreateTraceTailSamplingRules endpoint](/tooling/api-info/definition/operations/CreateTraceTailSamplingRules). > Requires [Chronoctl](/tooling/chronoctl) version 1.0.0 or later. You can use the `trace-tail-sampling-rules scaffold` command to generate an example tail sampling rule, and then copy the resource definition: ```shell theme={null} chronoctl trace-tail-sampling-rules scaffold ``` To define your tail sampling strategy: 1. Create a YAML file and define your tail sampling strategy. The following tail sampling drops all health check traces from an operation named `/health`. The `sample_rate` of `0` drops any traces matching the defined rule. ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: TraceTailSamplingRules spec: rules: - sample_rate: 0 name: Drop all health checks system_name: drop-node-health-checks filter: span: - operation: value: /health match: EXACT match_type: INCLUDE default_sample_rate: enabled: true sample_rate: 1 ``` 2. Apply your tail sampling strategy and send it to Observability Platform: ```shell /FILE_NAME/ theme={null} chronoctl apply -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of your tail sampling YAML file. Use Terraform to create a rule definition for tail sampling: 1. In your Terraform file, create a tail sampling strategy with a `resource` declaration by using the `chronosphere_trace_tail_sampling_rules` type, followed by a `default_sample_rate` and your `rules` definition. ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_trace_tail_sampling_rules" "default-sampling-rules" { default_sample_rate { enabled = true sample_rate = 1 } } ``` The `default_sample_rate` must be a value between `0` and `1`, inclusive. Refer to [these examples](#terraform-examples) for more information. 2. Apply the changes: ```terraform icon="square-terminal" theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`CreateTraceTailSamplingRules`](/tooling/api-info/definition/operations/CreateTraceTailSamplingRules) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Edit tail sampling rules Select from the following methods to edit tail sampling rules. To edit tail sampling rules with [Chronoctl](/tooling/chronoctl): 1. [View the tail sampling rules Chronoctl YAML](#view-tail-sampling-rules). 2. Modify its properties and apply the changes with the same process as [creating tail sampling rules](#create-tail-sampling-rules). Chronoctl updates tail sampling rules if it has the same slug. You can also use the following process if you already have a definition file: 1. Update the tail sampling rules definition file. 2. Run the following command to submit the changes: ```shell /FILE_NAME/ theme={null} chronoctl trace-tail-sampling-rules update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. To edit tail sampling rules using [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file that updates the resource's existing properties. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`UpdateTraceTailSamplingRules`](/tooling/api-info/definition/operations/UpdateTraceTailSamplingRules) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete tail sampling rules Select from the following methods to delete tail sampling rules. To delete a tail sampling rule with [Chronoctl](/tooling/chronoctl), use this command: ```shell /SLUG/ theme={null} chronoctl trace-tail-sampling-rules delete SLUG ``` Replace *`SLUG`* with the slug of the tail sampling rule you want to delete. For example, to delete a tail sampling rule with the slug `tail-sampling-prod`: ```shell theme={null} chronoctl trace-tail-sampling-rules delete tail-sampling-prod ``` To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteTraceTailSamplingRules`](/tooling/api-info/definition/operations/DeleteTraceTailSamplingRules) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Terraform examples Use the following examples to build your tail sampling strategy in Terraform. Because the tracing backend evaluates rules in match order, put expansive rules at the top of your Terraform file, such as rules that always drop or always keep specific traces. ### Default sampling rate The following example defines a default sample rate of `1`, which keeps all traces. A sample rate of `0` drops all traces that don't match any other rule. ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_trace_tail_sampling_rules" "default-sampling-rules" { default_sample_rate { enabled = true sample_rate = 1 } } ``` ### Drop all health check traces You might have load balancers that ping your backend servers every few seconds, which can generate a large amount of useless tracing data. In this instance, you can define a rule to drop all health check traces rather than those from a particular service. In addition to defining the default sample rate, the following rule drops all health check traces from an operation named `"/health"`. The `sample_rate` of `0` drops any traces matching the defined rule. ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_trace_tail_sampling_rules" "drop-node-health-checks" { default_sample_rate { enabled = true sample_rate = 1 } rules { name = "No Health Checks" system_name = "no_health_checks" filter { span { match_type = "INCLUDE" operation { match = "EXACT" value = "/health" } } } sample_rate = 0 } } ``` ### Always keep query traces with a minimum duration Requests to your app can quickly consume your licensed trace capacity. For example, user-initiated requests to a ride sharing app can amount to huge traces, especially during peak travel hours. Any time a query executes, it can generate tens or even hundreds of thousands of spans. You might only want to keep traces that exceed a specific duration or result in an error state, rather than storing the entirety of your tracing data. The following example keeps any trace with a span where the operation is `"/hail-ride"`, and the overall duration of the trace is greater than five seconds. This rule lets you store long-running traces and investigate what's causing higher latency. ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_trace_tail_sampling_rules" "keep-longer-traces" { default_sample_rate { enabled = true sample_rate = 1 } rules { name = "Hail Ride High Latency" system_name = "hail_ride_high_latency" sample_rate = 1 filter { span { match_type = "INCLUDE" operation { match = "EXACT" value = "/hail-ride" } } trace { duration { min_secs = 5 } } } } } ``` You can extend this rule set to also include traces to the `"/hail-ride"` operation that fail. The following rule matches any trace with at least one call to the `"/hail-ride"` operation anywhere in the trace, even if there's only one out of 1,000 spans. Observability Platform then keeps any traces from the `"/hail-ride"` operation where the error value is `true`. ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_trace_tail_sampling_rules" "keep-error-traces" { default_sample_rate { enabled = true sample_rate = 1 } rules { name = "Non-200 HTTP status, USA only" system_name = "non_200_http_status_usa" sample_rate = 1 filter { span { match_type = "INCLUDE" operation { match = "EXACT" value = "/hail-ride" } } trace { error { value = true } } } } } ``` ### Match on services in specific regions You might want to keep a percentage of traces from particular services that match certain conditions. For example, always keep a sample of traces from the `billing-svc` service in the `us-east` or `us-west` regions that have a specific duration. This ability to hone your sampling rules provides finer control over which tracing data you keep and pay for. The following example defines a `resource` definition with specified rules that matches two tags: * Matching a tag where the key is `region` and the values are either `us-east` or `us-west`. The example uses the `REGEX` operator to match either of the specified values. * Matching a tag where the key is `http.status_code` and the value doesn't match `200`. The example uses the `NOT_EQUAL` comparison operator to achieve this evaluation. Observability Platform applies the `sample_rate` of `0.6` to any traces matching that key-value pair and the additional specified criteria, such as `duration`, `error`, `operation`, and `service`. ```terraform expandable Terraform example icon="square-terminal" theme={null} resource "chronosphere_trace_tail_sampling_rules" "my-tail-sampling-rules" { default_sample_rate { enabled = true sample_rate = 0.5 } rules { name = "Non-200 HTTP status, USA only" system_name = "non_200_http_status_usa" filter { span { match_type = "INCLUDE" tag { key = "region" value { match = "REGEX" value = "(us-east|us-west)" } } tag { key = "http.status_code" numeric_value { comparison = "NOT_EQUAL" value = "200" } } duration { max_secs = 16 min_secs = 11 } error { value = true } operation { match = "EXACT" value = "execute-charge" } parent_operation { match = "EXACT" value = "execute-purchase" } parent_service { match = "EXACT" value = "purchase-svc" } service { match = "EXACT" value = "billing-svc" } span_count { min = 2 max = 4 } } trace { duration { min_secs = 10 max_secs = 15 } error { value = false } } } sample_rate = 0.6 } } ``` ### Nested tail sampling rules You can nest tail sampling rules by adding multiple `rules` definitions. The following example includes individual rules that match on different tags: * The `Reduce prod to 5 percent` rule matches a tag where the key is `BillingEnvironment` and the value is `production`. The sample rate is `0.05`, which samples 5% of traces matching this rule. * The `Exclude API status traces` rule matches a tag where the key is `Operation` and the value is `/api/status`. The sample rate is `0`, which drops all traces matching this rule. If traces match neither of these rules, Observability Platform applies the default rule, which is to keep all traces. ```terraform expandable Terraform example icon="square-terminal" theme={null} resource "chronosphere_trace_tail_sampling_rules" "two-tail-sampling-rules" { default_sample_rate { enabled = true sample_rate = 1 } rules { name = "Reduce prod to 5 percent" system_name = "reduce_prod_to_5percent" sample_rate = 0.05 filter { span { match_type = "INCLUDE" tag { key = "BillingEnvironment" value { match = "EXACT" value = "production" } } } } } rules { name = "Exclude API status traces" system_name = "exclude_api_status_operations" sample_rate = 0 filter { span { match_type = "INCLUDE" tag { key = "Operation" value { match = "EXACT" value = "/api/status" } } } } } } ``` # Trace Metrics Source: https://docs.chronosphere.io/control/shaping/sample-traces/trace-metrics Learn about how to create and use trace metrics to help address issues. Use Trace Explorer to identify the root cause of an issue. As you define your query and determine what caused the problem you're investigating, you can create a *trace metric* from your query to display that metric in a dashboard for faster discovery. You can then configure monitors and alerts on that metric to notify other on-call engineers if that metric identifies anomalies in your trace data. This ability to navigate from a trace metric to a specific query in Trace Explorer helps other on-call engineers address problems faster by using predefined queries to explore issues. Use the following metric types to help track information for collected traces: * **Counter metric**: Increments any time a trace that matches a filter gets collected. This metric also includes a label describing whether or not the trace contains an error. * **Histogram metric**: Collects information about the distribution of request durations for traces that match a filter. Chronosphere Observability Platform provides a default **Trace Metrics** dashboard with panels that display the requests, errors, and durations of requests associated with a created trace metric. The dashboard also includes a service map of the requests. Observability Platform generates any trace metrics you've created before applying behaviors or [tail sampling rules](/control/shaping/sample-traces/tail-sampling), which determine whether to persist the trace. To explore tracing data in Observability Platform, you must either install and configure the [Chronosphere Collector](/ingest/metrics-traces/collector/install/tracing) or the [OpenTelemetry Collector](/ingest/metrics-traces/otel/otel-ingest) to receive trace data from your services. ## View existing trace metrics You can view trace metrics in Observability Platform and open the related query in Trace Explorer. To return a list of defined trace metrics without additional information, use Chronoctl. Viewing a trace metric opens the related query in Trace Explorer. Observability Platform takes the contextual information in metrics from a dashboard and uses it to build links to search for traces. Clicking the link opens Trace Explorer and replaces the variables with matching criteria defined in the link. To view your existing trace metrics: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Metrics**. 2. In the **Rule Name** column, click the name of a trace metric to view the dashboard for that metric. 3. From the dashboard, click anywhere in one of the graph panels to pin that selection. In the pinned popup, click the and then click the three vertical dots icon and choose one of these options: * **Analyze anomaly (DDx)** opens [Metrics Differential Diagnosis (DDx)](/investigate/analyze/differential-diagnosis/metrics) and identifies labels and values that strongly correlate to the trace metric. * **View similar logs** opens Logs Explorer with results filtered to the selected trace metric. * **View similar traces** opens Trace Explorer with results filtered to the selected trace metric. See [common panel elements](/observe/dashboards/panels#common-panel-elements) for an explanation of the other available tools. To view a list of all available trace metrics using [Chronoctl](/tooling/chronoctl), use the `chronoctl trace-metrics-rules list` command: ```shell theme={null} chronoctl trace-metrics-rules list ``` A complete list of the available trace metrics displays in your terminal. To view a single trace metrics rule, use the `chronoctl trace-metrics-rules read` command, followed by the slug of the rule you want to view. For example, the following command returns a trace metrics rule with the slug `telemetry-sdk`: ```shell theme={null} chronoctl trace-metrics-rules read telemetry-sdk ``` To complete this action with the Chronosphere API, use the [`ListTraceMetricsRules`](/tooling/api-info/definition/operations/ListTraceMetricsRules) endpoint. To view a single trace metrics rule instead of a complete list of rules, use the [`ReadTraceMetricsRule`](/tooling/api-info/definition/operations/ReadTraceMetricsRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create a trace metric Observability Platform uses trace queries as the basis for trace metrics. You define a search query in Trace Explorer, and create a trace metric based on the matching criteria. You can search for any labels you add to your trace metric in [Metrics Explorer](/investigate/querying/metrics/explorer) when investigating query requests and responses. Creating a trace metric is the same as creating a new metric in your system. Any trace metrics you create impact your metrics license consumption, which display in the [License Overview](/observe/dashboards/managed-dashboards#licensing-dashboards). Because of that potential impact, avoid including high cardinality dimensions in your trace metric. As a protective measure, trace metrics capture data for only the first 1,000 unique series observed. Selecting a dimension to group by with high cardinality can overflow the 1,000 value trace metric cardinality limit, and cause incorrect data to display in your trace metric dashboard. To create a trace metric, you must have administrative privileges. To create a new trace metric: 1. In the navigation menu select ** Explorers > Trace Explorer**. 2. Apply one or more filters to [define a search](/investigate/querying/traces#define-a-search). Trace Explorer applies the filters represented in the query before updating metrics for the created trace metric. You can group your trace query search results by up to [three dimensions](/investigate/querying/traces#group-and-narrow-results) that can include service, operation, and any other span tag. Dimensions to group by are inherited from the defined search query. Trace metrics don't support regular expression operators, such as the match (`=~`) or doesn't match (`!~`) operators. 3. Click the three vertical dots icon, and then select **Create Trace Metric**. 4. In the **Create trace metric** dialog, enter a display name (the equivalent of Prometheus metric `__name__`) and system name (the trace metric **Rule Name**) for your trace metric. 5. In the **Group by labels** section, enter a metric label to display the aggregated results of your query grouped by that attribute. For each dimension in the **Group by labels** field, you must define a metric label key. Use the same key as the span tag, or a key that aligns with your existing metrics data. Observability Platform adds metric label keys to your trace metric, and associates the label value with the span tag. For example, if you group your query by the span tag **Service** and enter **container\_service** as the label key, Observability Platform adds a label to your trace metric where the label values equal the span tag **Service** values. You can then query by that label anywhere you search for metric labels, such as in Metrics Explorer or . 6. Optional: Expand the **Static metric labels** section and enter key-value pairs to add static labels to your trace metric. Static labels are like metadata you add to your trace metric. 7. Click **Create**. After creating the trace metric, the **Trace Metric Created** dialog displays. It can take several minutes for the dashboard to display data about the trace metric. 8. Click **Go to Trace Metrics** to view the list of available trace metrics. When your trace metric is available, you can select it from this list to view a dashboard for the trace metric that includes requests, errors, durations, and a topology map. 9. When viewing your trace metric, click a data point in any of the provided graphs and then click **Query Traces** to open Trace Explorer with the query you defined for your trace metric. When creating your trace metric with Chronoctl, consider the following behaviors of the `slug` property: * If you don't provide a `slug`, Chronoctl generates one based on the `name` field. * You can modify the `name` after creating your trace metric, but can't modify the `slug`. * The `metric_name` and `slug` must match. To use [Chronoctl](/tooling/chronoctl) to create a trace metric, use the `chronoctl trace-metrics-rules` command: ```shell theme={null} chronoctl trace-metrics-rules scaffold ``` 1. Run the following command to generate a sample trace metric configuration you can use as a template: ```shell theme={null} chronoctl trace-metrics-rules scaffold ``` In the template, `kind: TraceMetricsRule` defines an individual trace metric. You can redirect the results (using the redirection operator `>`) to a file for editing. 2. With a completed definition, submit it with: ```shell /FILE_NAME/ theme={null} chronoctl trace-metrics-rules create -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. See the [Chronoctl example](#chronoctl-trace-metric-example) for a completed trace metrics resource. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. To create a new trace metric with [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file and add the definition by using the `chronosphere_trace_metrics_rule` type, followed by a name in a resource declaration. Refer to [Trace metric rule fields](#trace-metric-rule-fields) for a list of all supported fields. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` See the [Terraform example](#terraform-trace-metric-example) for a completed trace metric resource. To complete this action with the Chronosphere API, use the [`CreateTraceMetricsRule`](/tooling/api-info/definition/operations/CreateTraceMetricsRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Chronoctl trace metric example The following resource defines a trace metric that matches spans that includes both the `ordering-svc` and the `POST/submit-order` operation, a tag of `env=production`, and a duration between two and three seconds. The metric also groups by the `ordering-svc` key. Use the `scope_filter` field to add a secondary layer of filtering after the initial trace metrics filter. This additional filtering is comparable to using the [span statistics](/investigate/querying/traces/features#span-statistics) feature in Trace Explorer to return information about specific spans within the specified trace filter. ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: TraceMetricsRule spec: name: Ordering service uswest1 error traces slug: ordering-service-uswest1-error-traces metric_name: order-svc-uswest1-errors group_by: - label: svc key: named_key: ordering-svc type: SERVICE trace_filter: span: - tags: - key: env value: value: production match: EXACT operation: value: POST/submit-order match: EXACT service: value: ordering-svc match: EXACT trace: duration: min_secs: 2 max_secs: 3 error: value: true scope_filter: span_scopes: - service: value: ordering-svc match: EXACT match_type: INCLUDE - operation: value: /ordering.Ordering/Checkout match: EXACT - duration: min_secs: 2 ``` ### Terraform trace metric example The following Terraform resource definition defines a trace metric that matches spans that include both the `ordering-svc` and the `POST/submit-order` operation, a tag of `env=production`, and a duration between two and three seconds. The metric also groups by the `ordering-svc` key. Use the `scope_filter` field to add a secondary layer of filtering after the initial trace metrics filter. This additional filtering is comparable to using the [span statistics](/investigate/querying/traces/features#span-statistics) feature in Trace Explorer to return information about specific spans within the specified trace filter. ```terraform expandable Terraform example icon="square-terminal" theme={null} resource "chronosphere_trace_metrics_rule" "ordering_service_metric" { name = "Ordering service uswest1 error traces" metric_name = "order-svc-uswest1-errors" group_by { label = "svc" key { type = "SERVICE" named_key = "ordering-svc" } } trace_filter { span { tag { key = "env" value { value = "production" match = "EXACT" } } operation { value = "POST/submit-order" match = "EXACT" } service { value = "ordering-svc" match = "EXACT" } } trace { duration { min_secs = 2 max_secs = 3 } error { value = true } } scope_filter { span_scopes { service { value = "ordering-svc" match = "EXACT" } match_type = "INCLUDE" } } } } ``` ## Update a trace metric Select from the following methods to update trace metrics. To update a trace metric, you must have administrative privileges. To edit or update a trace metric: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Metrics**. 2. To the right of the date of the trace metric, click the three vertical dots icon, and then click **Edit**. 3. In the **Edit trace metric** dialog, edit the query that generates the trace metric or make any other needed changes. 4. Click **Save** to save your changes. To update a trace metric using [Chronoctl](/tooling/chronoctl): 1. Return a list of all available trace metrics: ```shell theme={null} chronoctl trace-metrics-rules list ``` 2. Modify the YAML file for the trace metrics you want to update. 3. Apply your changes: ```shell theme={null} chronoctl apply -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of your trace metric YAML file. Chronoctl updates the trace metric's properties if it has the same slug. To edit a trace metric using [Terraform](/tooling/infrastructure/terraform): 1. Modify the resource's existing properties. 2. Run `terraform apply` to apply the changes. ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`UpdateTraceMetricsRule`](/tooling/api-info/definition/operations/UpdateTraceMetricsRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete a trace metric Select from the following methods to delete trace metrics. To delete a trace metric, you must have administrative privileges. To delete a trace metric: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Trace Metrics**. 2. To the right of the date of the trace metric, click the three vertical dots icon, and then click **Delete**. To delete a trace metric using [Chronoctl](/tooling/chronoctl): 1. Return a list of all available notification policies: ```shell theme={null} chronoctl trace-metrics-rules list ``` 2. Run the following command to delete the trace metric: ```shell theme={null} chronoctl trace-metrics-rules delete SLUG ``` Replace *`SLUG`* with the slug of the trace metric you want to delete. Chronoctl deletes the trace metric with the specified slug. To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteTraceMetricsRule`](/tooling/api-info/definition/operations/DeleteTraceMetricsRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. Observability Platform removes the metric from the **Trace Metrics** page. Deleting a trace metric also removes access to the metric in the Trace Metrics dashboard. ## Trace metric rule fields A trace metric rule consists of the following fields, which are properties of the `trace_metrics_rule` object. Each field is required unless otherwise noted. See the [CreateTraceMetricsRule](/tooling/api-info/definition/operations/CreateTraceMetricsRule) API for a complete list of supported fields. * `group_by`: Labels for grouping and narrowing search results to specific attributes. See [Group and narrow results](/investigate/querying/traces#group-and-narrow-results) for more information about grouping related attributes. * `key`: The key to group by. * `named_key`: The name of the key to group by. * `type`: The type of key to group by. * `label`: The dimension for displaying the aggregated results of your query grouped by that attribute in resulting trace metrics. Defaults to the selected service. * `histogram_buckets_seconds`: Optional: An array of custom buckets measured in seconds for duration histogram metrics. You can set these if the default buckets aren't appropriate for your data. The default buckets are `[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]`. * `metric_labels`: Optional: Static labels applied to the metrics, defined as an object of one or more label name-value pairs. * `metric_name`: The metric's name as stored in the metrics database. Use this name to refer to the metric in PromQL queries, such as in dashboards, monitors, and the Metrics Explorer. The `metric_name` must follow [Prometheus metric naming rules](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). Although you can modify the `metric_name` after creation, any data points generated *before* the name change aren't renamed. The new metric name is instead used to generate new data points starting from that point in time. The old metric name doesn't add new data, but you can still query it for past data points. * `name`: The name of the trace metric rule. You can modify this name after creating the trace metric rule. * `slug`: Optional: The slug for the trace metric rule. After you create the rule, the slug is immutable. If omitted, Observability Platform generates a `slug` at creation time that's based on the `name`. * `trace_filter`: A filter object that evaluates traces against several criteria and emits metrics for only those that match. The filter's capabilities are similar to the filter in the [Trace Explorer](/investigate/querying/traces), except it supports only exact matches on string fields instead of regular expressions. This object supports additional [optional object fields](#optional-trace-filter-object-fields). ### Optional trace filter object fields The required `trace_filter` object supports the following optional fields, and you can use any combination of them: Unlike most Terraform resources, several child fields of `trace` and `span` are objects with a single field named `value`. * `trace`: Applies trace-level filtering. You can specify only one trace filter. * `duration`: Matches traces with a specified duration. You can specify only one of `min_secs` and `max_secs`. * `min_secs`: Matches traces with a duration greater than or equal to this value. * `max_secs`: Matches traces with a duration less than or equal to this value. * `error`: Object with a single field named `value`. If specified, matches traces with an error flag equal to this Boolean value. Refer to [assign values with `value`](#assign-values-with-value) for more information. * `span`: Applies span-level filtering. You can specify multiple span filters, but Observability Platform evaluates a trace filter match only if *all* `span` filters match. * `service`: Object with a single field named `value`. Matches spans with a service field equal to this value. Refer to [assign values with `value`](#assign-values-with-value) for more information. * `operation`: Object with a single field named `value`. Matches spans with an operation field equal to this value. Refer to [assign values with `value`](#assign-values-with-value) for more information. * `parent_service`: Object with a single field named `value`. Matches spans whose parent span's service field is equal to this value. Refer to [assign values with `value`](#assign-values-with-value) for more information. * `parent_operation`: Object with a single field named `value`. Matches spans whose parent span's operation field is equal to this value. Refer to [assign values with `value`](#assign-values-with-value) for more information. * `duration`: Matches spans with a specified duration. You can specify only one of `min_secs` and `max_secs`. * `min_secs`: Matches spans with a duration greater than or equal to this value. * `max_secs`: Matches spans with a duration less than or equal to this value. * `error`: Object with a single field named `value`. If specified, matches traces with an error flag equal to this Boolean value. Refer to [assign values with `value`](#assign-values-with-value) for more information. * `tags`: Matches based on a span's tag keys and values. You must specify the `key`, and you can specify multiple tag filters, but Observability Platform evaluates a span filter match only if *all* `tag` filters match. * `key`: Matches spans whose tags match the specified key. * `value`: Object with a single field named `value`. Matches spans when the tag with a matching key also matches the specified value. If omitted, Observability Platform evaluates *all* spans matching the key as matches. Refer to [assign values with `value`](#assign-values-with-value) for more information. * `span_count`: Specifies how many spans can match. You can set only one of `min` and `max`. By default, Observability Platform evaluates a `span` filter as a match only if at least one span matches. * `min`: If set, *at least* this number of spans must match the parent span filter's conditions for Observability Platform to evaluate the entire trace filter as a match. * `max`: If set, Observability Platform evaluates a trace filter match only if the number of spans matching the parent span filter's conditions is equal to or less than the `max` value. * `match_type`: Specifies the span filter's match type. Valid values are `"include"` and `"exclude"`. The `"include"` match type is the default, and evaluates a trace filter match if *all* its fields match *any* span, or multiple spans if you specify a `span_count`. An `"exclude"` match type evaluates a trace filter match if *no* spans within that trace match *all* of the span filter's conditions. ### Assign values with `value` If a child field of a `trace` or `span` are objects with a single field named `value`, assign the value to the required child `value` field of these objects, instead of directly assigning the value like a field. For example, to set the `error` field's value in a `trace` filter, use: ```terraform Terraform example icon="square-terminal" theme={null} trace { error { value = true } } ``` Also, the value of a `tag` filter, unlike the key, is also an object with a `value` field: ```terraform Terraform example icon="square-terminal" theme={null} span { tag { key = "region" value = { value = "us-east" } } } ``` Although these objects have only the one `value` field, this structure lets Chronosphere plan future features for these criteria. For more examples of this structure, see the [filter examples](#filter-examples). ### Filter examples Matches traces marked as `error` that also took more than five seconds: ```terraform Terraform example icon="square-terminal" theme={null} trace_filter { trace { duration { min_secs = 5 } error { value = true } } } ``` Matches traces with at least one span from the service named `"cupcake-factory"`: ```terraform Terraform example icon="square-terminal" theme={null} trace_filter { span { service { value = "cupcake-factory" } } } ``` Matches traces with at least one span tagged `region:us-east` *and* at least one span tagged `region:us-west`: ```terraform Terraform example icon="square-terminal" theme={null} trace_filter { span { tag { key = "region" value = { value = "us-east" } } } span { tag { key = "region" value = { value = "us-west" } } } } ``` Matches traces with at least one span tagged *both* `region:us-east` *and* `stack:production`: ```terraform Terraform example icon="square-terminal" theme={null} trace_filter { span { tag { key = "region" value = { value = "us-east" } } tag { key = "stack" value = { value = "production" } } } ``` Matches traces with *at least* 10 spans containing the `"db-query"` operation: ```terraform Terraform example icon="square-terminal" theme={null} trace_filter { span { operation { value = "db-query" } span_count { min = 10 } } } ``` # Shape your log data Source: https://docs.chronosphere.io/control/shaping/shape-logs Learn how to use different tools in Chronosphere Observability Platform to shape your log data and help control costs. After creating, monitoring, and adjusting [budgets](/control/consumption/budgeting) for your log data, you can use tools in Chronosphere Observability Platform to transform, reshape, retain, or exclude data. Managing the data you persist can reduce your overall consumption, provide predictability in generated log data, and result in more performant queries when searching through logs. After sending data to Observability Platform, apply logic to parse, normalize, control, and format your logs before persisting the data to the database. Parsing logic and field normalization are applied before any control rules run. ```mermaid actions={false} theme={null} flowchart LR Send["Send data"] subgraph OP["Observability Platform"] Parse[Parse data] Normalize[Normalize data] Control[Control data] Format[Format data] Store[Store data] end Send --> Parse --> Normalize --> Control --> Format --> Store click Parse "/control/shaping/shape-logs/parse-logs" click Normalize "/control/shaping/shape-logs/normalize-logs" click Control "/control/shaping/shape-logs/control-logs" click Format "/control/shaping/shape-logs/lookup-logs/" classDef stages fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF; class Parse,Normalize,Control,Format stages ``` * **Parse**: To create structured logs from plaintext logs, and parse individual fields to extract meaningful data without changing your instrumentation, use [parsing](/control/shaping/shape-logs/parse-logs). * **Normalize**: To apply custom field mappings to normalize log data, use [field normalization](/control/shaping/shape-logs/normalize-logs). * **Control**: To drop entire logs, drop individual fields, or sample a percentage of your log data, create [control rules](/control/shaping/shape-logs/control-logs). * **Format**: To add context and enrich logging queries with static values, use [lookup tables](/control/shaping/shape-logs/lookup-logs). # Control your log data Source: https://docs.chronosphere.io/control/shaping/shape-logs/control-logs Use log control rules to define actions for a subset of your log data that match specific criteria. Chronosphere Observability Platform provides control rules that let you manage log data by transforming, reshaping, retaining, or excluding data before it's stored. These rules let you control the volume of persisted log data and improve data quality. By controlling which data you keep or drop, you can reduce noise, better control costs, and focus on data that matters most to your business. ## Applying control rules Control rules run after both [parsers](/control/shaping/shape-logs/parse-logs) and [field mappings](/control/shaping/shape-logs/normalize-logs) run. Disabled rules are skipped. [Enable rules](#enable-or-disable-control-rules) to have Observability Platform evaluate them. Observability Platform applies control rules serially and in order. Each rule operates on the output of the rule before it. Effects of previous rules apply immediately to the next rule in the sequence. If a rule matches the same condition as a previous rule, the subsequent rule operates only on the outcome of the previous rule. For example, if the first rule samples logs by 50% where `service = "auth"` and the second rule also includes this condition, the second rule evaluates only the remaining 50% of the original data volume. ```mermaid actions={false} theme={null} flowchart LR Send["Send data"] subgraph OP["Observability Platform"] Parse[Parse data] Normalize[Normalize data] Control[Control data] Format[Format data] Store[Store data] end Send --> Parse --> Normalize --> Control --> Format --> Store click Parse "/control/shaping/shape-logs/parse-logs" click Normalize "/control/shaping/shape-logs/normalize-logs" click Format "/control/shaping/shape-logs/lookup-logs/" classDef stages fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF; class Control stages ``` Observability Platform includes the following control rules: * **Drop logs** that match a specific filter to remove unnecessary or low-value data. * **Drop fields** from logs that match specified conditions, which helps reduce the size of individual logs. If a log contains no data after fields are dropped, you can choose to drop the entire log. * **Sample logs** to retain a certain percentage of matching logs and drop the remainder, which can reduce volume and maintain visibility. * **Emit metric** converts log data to metric data. Use this control rule when you want to reduce the volume of logs, retain data for a longer period, and increase the speed of querying and filtering for metrics instead of logs. * **Replace field** identifies a specified field in your log data and uses a regular expression to replace any matches based on the selected replacement mode. Use this control rule to truncate long fields and preserve your log structure. * **Parse field** formats and splits unstructured fields into multiple fields, conditionally parses logs with similar fields based on a source field, and remaps fields. To change the order that control rules are applied, [reorder rules](#reorder-control-rules). ## View control rules Select from one of the following methods to view control rules. To view control rules: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. All defined control rules display in the **Control rules** section. 2. To view only control rules, from the **View** dropdown, select **Control rules**. 3. To view the code definition for all configured control rules, click the **Code config** tab. The **Logs control rules** section includes all defined control rules. To use [Chronoctl](/tooling/chronoctl) to return all control rules, use the `chronoctl log-control-config read` command: ```shell theme={null} chronoctl log-control-config read ``` [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) in Observability Platform to view a control rule's Terraform representation. To complete this action with the Chronosphere API, use the [`ReadLogControlConfig`](/tooling/api-info/definition/operations/ReadLogControlConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create control rules Use control rules to drop certain logs at ingestion, or fields that you don't want to persist. For example, dropping log data from a specific cluster, test data in a development environment, or unnecessary individual fields across your logs. When creating control rules, be sure you understand how Observability Platform [applies control rules](#applying-control-rules). Use one of the following methods to create control rules for log data: Define control rules in Observability Platform, and then [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the rule definition. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the **Control rules** section, click **Create**. 3. In the **Create control rule** page, enter a name for the control rule. 4. From the **Action** menu, select the type of rule to create: * **Drop logs**: Completely drop logs that match the specified filter. * **Drop fields**: Remove specific fields from logs that match the specified filter. * **Sample logs**: Retain a certain percentage of matching logs and drop the remainder. * **Emit metric**: Convert log data to metric data. * **Replace field**: Replace context within a log field using a regular expression. * **Parse field**: Structure, format, or separate fields out of a source field. 5. Enter a filter to return log data for the control rule, and then press `Ctrl+Enter` (`Command+Return` on macOS) to submit the filter. This rule applies only to logs that match this filter at the time the log data was ingested. 6. Review the returned data from the preview filter and make changes as necessary. Control rules are applied during ingestion, so the preview filter results might differ from a filter submitted in Logs Explorer after ingestion. By default, including a dot (`.`) in a filter indicates nesting within a field. In some instances, fields with a key that uses dot notation within the name, such as `resource.type`, must be surrounded by double quotes and brackets in the control rule filter. If a warning message displays indicating that `This filter is applied at ingestion` and the filter returns no results, wrap the key in double quotes ( `"`) and brackets (`[]`). For example, the following filter contains the `resource.type` key, which requires surrounding the field in double quotes and brackets for this particular service: ```text theme={null} service = "nginx" and severity = "INFO" and ["resource.type"] = "k8s_container" ``` 7. Enter additional information for the specified action: Review the filter and make changes as needed. Define the criteria for the drop rule: * In the **Parent path** field, enter the level of depth in the log to apply the regular expression to. For example, if you wanted to apply the regular expression to fields under `httpRequest`, such as `httpRequest.status`, enter `httpRequest` in the **Parent path** field. For fields with no nesting, leave the **Parent path** field empty. * In the **Field regex** field, enter the regular expression that determines which fields to match on. In the **Logs to keep** field, enter a percentage of logs you want to keep. For example, `25` for 25%. Define the criteria for the rule to emit metrics from logs: * Enter a unique name for the generated metric. * Select the metric mode for the generated metric: * **Counter**: A metric that counts the number of times a certain event has occurred. * **Gauge**: A metric that tracks a value which changes over time. Choose **Last** (most recent), **Min** (minimum value), or **Max** (maximum value) to define which value to use. * **Exponential Histogram**: A metric that tracks the distribution of values across data groupings with exponentially increasing size. * Enter a name for the log field whose value you want to use as the metric value. Use `field[field]` syntax to indicate nested fields. * Enter labels as key-value pairs to add to the metric. * To drop the original log, select the **Drop original log after emitting metric** checkbox. Define the criteria for the replace rule: * Enter the name of the log field to apply the rule to. In the **Match regex** field, enter the regular expression that determines which data to match on in the specified field. * Determine whether to replace the first match only (**Replace first**), or to replace all matches (**Replace all**). * Select the replacement mode: * **Static value**: Replace the matched content with a static string. For example, replace punctuation in a field with an empty string, or truncate the ends of long stack traces. * **Mapped value**: Replace the matched content with specified key-value pairs. For example, reduce log volume by replacing a string error with a specific error code. If none of the key-value pairs match, the provided default value is used. Click **Provide default value** to specify a default value if no matching key is found. * **Hash**: Replace the matched content with a hashed string, which can help reduce the size of large strings. After replacing the original content with a string, there's no way to recover that information. Define the criteria to parse logs: * Enter a **Source key** and optional **Destination key**. The source key is the field in the log whose string value you want to parse, such as `message`. The destination key is the field where the parsed output is written back into the log, such as `message.parsed`. If you don't enter a destination key, any parsed fields are pulled into the top level of the log. * Select the parser type to apply to the source key and destination key: * **JSON**: The parser reads the source field's string value as a JSON object. Keys and values in the JSON become fields in the output. * **Regex**: Use a regular expression to match logs. In the **Template** field, select the regular expression template to apply to your logs. Use a predefined template such as **Nginx**, or specify a **Custom** template. * **Grok**: Enter a Grok pattern to match logs. This option lets you enter pre-built Grok named patterns like `%{IP:client}`, `%{NUMBER:bytes}`, and `%{WORD:method}`. * **Key/value**: Use a specified pattern to separate key-value pairs in log data. Enter the following values: * **Field delimiter**: Specifies the character used to split the input log data into key-value pairs. For example, a comma (`,`) if your data is in the format `key: value, key: value`. * **Key/value separator**: Specifies the character used to split matching pairs into its key and value. For example, a colon (`:`) if your data is in the format `key: value`. * **Trim characters**: Identifies the leading and trailing characters to trim from the key-value pairs. For example, a space (`" "`) between key-value pairs if your data is in the format `key: value, key: value`. * To test the parser, in the **Preview field parser** section, enter a portion of your log payload in the **Input** field. For example: ```text wrap theme={null} 192.168.1.1 - john [10/Oct/2023:13:55:36 -0700] "GET /api/users HTTP/1.1" 200 1234 ``` * Click **Test parser** to view a preview of the generated output after the parser runs. For example, the following output is generated in the **Output** section from the input string in the previous step: ```json theme={null} { "code": "200", "host": "192.168.1.1", "method": "GET", "path": "/api/users HTTP/1.1", "size": "1234", "time": "10/Oct/2023:13:55:36 -0700", "user": "john" } ``` * Click **Save** to save the parser definition. * On the **Logs Ingest** page, click **View code config**. * Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. 8. Click **Save** to save the control rule definition. 9. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. To use [Chronoctl](/tooling/chronoctl) to create control rules, use the `chronoctl log-control-config create` command: ```shell theme={null} chronoctl log-control-config create ``` 1. Run the following command to generate a sample control rule configuration you can use as a template: ```shell theme={null} chronoctl log-control-config scaffold ``` In the template, `kind: LogControlConfig` defines an individual control rule configuration. 2. With a completed definition, submit it with: ```shell /FILE_NAME/ theme={null} chronoctl log-control-config create -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. See the [Chronoctl example](#chronoctl-example) for more information. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. To create a control rule with [Terraform](/tooling/infrastructure/terraform): 1. Create a Terraform resource with the `chronosphere_log_control_config` resource type, followed by a resource name. 2. Add the definition to a Terraform file. 3. Run this command to create the resource: ```shell theme={null} terraform apply ``` See the [Terraform example](#terraform-example) for more information. To complete this action with the Chronosphere API, use the [`CreateLogControlConfig`](/tooling/api-info/definition/operations/CreateLogControlConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Chronoctl example The following example defines a control rules file for Chronoctl, and contains the following rules: * The first rule samples 10% of logs that match the defined filter. * The second rule drops all logs for the `nginx` service of severity `INFO` where the `http.Request.status` equals `200`. * The third rule drops all `kubernetes` fields that match the defined filter. The regular expression indicates that any fields matching `kubernetes.[FIELD]` are dropped from all logs. * The fourth rule uses a mapped value to search for the `error_code` field, and replaces values based on the provided key-value pairs. If the rule matches on `INTERNAL_SERVER_ERROR`, it replaces that value with `500`. If the rule matches on `NOT_FOUND`, it replaces the value with `401`. * The fifth rule emits a counter metric from NGINX error logs. Each matching log increments `nginx_errors_total` by one. Labels on the metric come from the `service` and `error_code` log fields. * The sixth rule parses the `raw_request` field on matching NGINX access logs using a Grok pattern. The parsed fields are written to the `request` field. ```yaml expandable Chronoctl example icon="square-terminal" wrap lines highlight={6,17,22,35,52,68} theme={null} api_version: v1/config kind: LogControlConfig spec: # The ordered list of control rules. rules: # First rule - name: "Sample billing service with missing user ID" mode: ENABLED # Log filter syntax to select logs. Only matching logs will have control action applied. filter: "cluster_name:'production'\n AND service='billing-service'\n AND message = 'missing user ID'" type: SAMPLE sample: # Percentage of matching logs to keep. Must be in the range [0, 1]. rate: 0.10 # Second rule - name: "Drop NGINX information logs" mode: ENABLED filter: "service = 'nginx' and severity = 'INFO' and httpRequest.status = '200'" type: DROP # Third rule - name: 'Drop Kubernetes fields from development environments' mode: ENABLED filter: "namespace = 'development' and kubernetes.cluster_name = 'test'" # The configuration for a drop field action. type: DROP_FIELD drop_field: # Regular expression to match the field names to drop. field_regex: .* # Selector to indicate field path. Use 'parent[child]' syntax to # indicate nesting. parent_path: selector: kubernetes # Fourth rule - name: error replacement mode: ENABLED filter: service = "nginx" and severity = "ERROR" type: REPLACE_FIELD replace_field: field: selector: error_code replace_regex: .* replace_all: true replace_mode: MAPPED_VALUE mapped_value: pairs: - key: INTERNAL_SERVER_ERROR value: "500" - key: NOT_FOUND value: "401" # Fifth rule - name: "Emit counter for NGINX errors" mode: ENABLED filter: service = "nginx" and severity = "ERROR" type: EMIT_METRICS emit_metrics: # Prometheus-compatible metric name. name: nginx_errors_total mode: COUNTER labels: - key: service value: selector: service - key: error_code value: selector: error_code # Sixth rule - name: "Parse NGINX access log request field" mode: ENABLED filter: service = "nginx" and severity = "INFO" type: PARSE_FIELD parse_field: # The source field containing the raw request string to parse. source: selector: raw_request # The destination field where parsed structured data will be written. destination: selector: request parser: parser_type: GROK grok_parser: # Named capture groups become named fields in the extracted log. pattern: "%{WORD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:version}" ``` ### Terraform example The following example defines a control rules resource for Terraform, and contains the following rules: * The first rule samples 10% of logs that match the defined filter. * The second rule drops all logs for the `nginx` service of severity `INFO` where the `http.Request.status` equals `200`. * The third rule drops all `kubernetes` fields that match the defined filter. The regular expression indicates that any fields matching `kubernetes.[FIELD]` are dropped from all logs. * The fourth rule uses a mapped value to search for the `error_code` field, and replaces values based on the provided key-value pairs. If the rule matches on `INTERNAL_SERVER_ERROR`, it replaces that value with `500`. If the rule matches on `NOT_FOUND`, it replaces the value with `401`. * The fifth rule emits a counter metric from NGINX error logs. Each matching log increments `nginx_errors_total` by one. Labels on the metric come from the `service` and `error_code` log fields. * The sixth rule parses the `raw_request` field on matching NGINX access logs using a Grok pattern. The parsed fields are written to the `request` field. ```terraform expandable Terraform example icon="square-terminal" wrap lines highlight={4,24,32,51,82,110} theme={null} resource "chronosphere_log_control_config" "log_control_config" { # The ordered list of control rules. rules { # First rule name = "Sample billing service with missing user ID" # Log filter syntax to select logs. Only matching logs will have control action applied. filter = chomp(<<-EOF cluster_name:"rc" AND service="billing-service" AND message = "missing user ID" EOF ) mode = "ENABLED" sample { # Percentage of matching logs to keep. Must be in the range [0, 1]. rate = 0.01 } # The configuration for a drop field action. type = "SAMPLE" } rules { # Second rule name = "Drop NGINX information logs" filter = "service = \"nginx\" and severity = \"INFO\" and httpRequest.status = \"200\"" mode = "ENABLED" type = "DROP" } rules { # Third rule name = "Drop Kubernetes fields from development environments" drop_field { # Regular expression to match the field names to drop. field_regex = ".*" # Selector to indicate field path. Use 'parent[child]' syntax to # indicate nesting. parent_path { selector = "kubernetes" } } filter = "chronosphere_namespace = \"rc-logging\" and kubernetes.cluster_name = \"rc-b\"" mode = "ENABLED" type = "DROP_FIELD" } rules { # Fourth rule name = "error replacement" filter = "service = \"nginx\" and severity = \"ERROR\"" mode = "ENABLED" replace_field { field { selector = "error_code" } mapped_value { pairs { key = "INTERNAL_SERVER_ERROR" value = "500" } pairs { key = "NOT_FOUND" value = "401" } } replace_all = true replace_mode = "MAPPED_VALUE" replace_regex = ".*" } type = "REPLACE_FIELD" } rules { # Fifth rule name = "Emit counter for NGINX errors" filter = "service = \"nginx\" and severity = \"ERROR\"" mode = "ENABLED" type = "EMIT_METRICS" emit_metrics { # Prometheus-compatible metric name. name = "nginx_errors_total" mode = "COUNTER" labels { key = "service" value { selector = "service" } } labels { key = "error_code" value { selector = "error_code" } } } } rules { # Sixth rule name = "Parse NGINX access log request field" filter = "service = \"nginx\" and severity = \"INFO\"" mode = "ENABLED" type = "PARSE_FIELD" parse_field { # The source field containing the raw request string to parse. source { selector = "raw_request" } # The destination field where parsed structured data will be written. destination { selector = "request" } parser { parser_type = "GROK" grok_parser { # Named capture groups become named fields in the extracted log. pattern = "%%{WORD:method} %%{URIPATHPARAM:path} HTTP/%%{NUMBER:version}" } } } } } ``` ## Edit control rules When creating or editing control rules, you can [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to view code representations of the rules. The displayed code also responds to changes you make in the **Visual editor** tab. Select from the following methods to edit control rules. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the row of the control rule you want to edit, click the three vertical dots icon and then click **Edit control rule**. 3. In the **Edit control rule** drawer, make changes to your control rule, and then click **Save**. 4. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. Observability Platform saves changes to your control rule. To update control rules with [Chronoctl](/tooling/chronoctl), use the `log-control-config update` command: ```shell theme={null} chronoctl log-control-config update ``` 1. Update the control rule definition file. 2. Run the following command to submit the changes: ```shell /FILE_NAME/ theme={null} chronoctl log-control-config update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To edit control rules using [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file that updates the resource's existing properties. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` You can also [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to view the control rule's Terraform representation. To complete this action with the Chronosphere API, use the [`UpdateLogControlConfig`](/tooling/api-info/definition/operations/UpdateLogControlConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Reorder control rules Control rules are applied in descending order. To change the order control rules are applied, reorder rules. Disabled rules are skipped. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the **Control rules** section, click **Re-order rules**. 3. In the **Edit control rules** drawer, in the row of the control rule you want to move, click the drag indicator icon and move the control rule to a different order position. 4. Click **Save**. 5. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. Observability Platform saves changes to your control rules. To reorder control rules with [Chronoctl](/tooling/chronoctl), use the `log-control-config update` command: ```shell theme={null} chronoctl log-control-config update ``` 1. Update the control rule definition file. 2. Run the following command to submit the changes: ```shell /FILE_NAME/ theme={null} chronoctl log-control-config update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To reorder control rules using [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file that updates the resource's existing properties. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` You can also [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to view the control rule's Terraform representation. To complete this action with the Chronosphere API, use the [`UpdateLogControlConfig`](/tooling/api-info/definition/operations/UpdateLogControlConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Enable or disable control rules Each rule has a mode, which can be enabled or disabled. By default, any created rule is enabled. Any rule that's enabled is applied. Disabled rules are skipped. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the row of the control rule you want to enable or disable, click the toggle in the **Enabled** column. 3. Click **Save**. 4. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. Observability Platform saves changes to your control rules. To enable or disable control rules with [Chronoctl](/tooling/chronoctl), use the `log-control-config update` command: ```shell theme={null} chronoctl log-control-config update ``` 1. In the control rule definition file, for the rule you want to modify, change the current `mode`. For example, to disable a rule, change `mode: ENABLED` to `mode: DISABLED`. 2. Run the following command to submit the changes: ```shell /FILE_NAME/ theme={null} chronoctl log-control-config update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To enable or disable control rules using [Terraform](/tooling/infrastructure/terraform): 1. In the control rule Terraform file, change the current `mode` for the rule you want to modify. For example, to disable a rule, change its `mode = ENABLED` to `mode = DISABLED`. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` You can also [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to view the control rule's Terraform representation. To complete this action with the Chronosphere API, use the [`UpdateLogControlConfig`](/tooling/api-info/definition/operations/UpdateLogControlConfig) endpoint. In the control rule definition, change the current `"mode"` for the rule you want to modify. For example, to disable a rule, change its `"mode": "ENABLED"` to `"mode": "DISABLED"`. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete control rules Select from the following methods to delete control rules. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the row of the control rule you want to delete, click the three vertical dots icon and then click **Delete control rule**. 3. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. Observability Platform deletes your control rule. To delete a control rule with [Chronoctl](/tooling/chronoctl), use the `chronoctl log-control-config delete` command: ```shell theme={null} chronoctl log-control-config delete ``` Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteLogControlConfig`](/tooling/api-info/definition/operations/DeleteLogControlConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # Create lookup tables for log data Source: https://docs.chronosphere.io/control/shaping/shape-logs/lookup-logs Learn how to create and modify lookup tables for log data in Chronosphere Observability Platform. When [filtering log data](/investigate/querying/query-logs), you want to quickly retrieve results, especially when debugging an issue or responding to an incident. To help provide more meaningful results, create lookup tables in a spreadsheet (in CSV format) and upload them to Chronosphere Observability Platform. Lookup tables store pre-calculated results or mappings between values, allowing for faster data retrieval. Use lookup tables to add context, normalize values, and enrich log data for improved filtering. For example, you can enrich severity fields with explanations, map keys like error codes to human-readable descriptions, and standardize inconsistent data. Lookup tables help you apply consistent mappings and enrichments within queries, without hard-coding large or repetitive mapping logic inline. ```mermaid actions={false} theme={null} flowchart LR Send["Send data"] subgraph OP["Observability Platform"] Parse[Parse data] Normalize[Normalize data] Control[Control data] Format[Format data] Store[Store data] end Send --> Parse --> Normalize --> Control --> Format --> Store click Parse "/control/shaping/shape-logs/parse-logs" click Normalize "/control/shaping/shape-logs/normalize-logs" click Control "/control/shaping/shape-logs/control-logs" classDef stages fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF; class Format stages ``` After uploading lookup tables, users can reference them in log queries using the [`lookup` function](/investigate/querying/query-logs/query-syntax#lookup). ## View lookup tables To view lookup tables: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Lookup Tables**. All lookup tables are displayed. 2. To view a lookup table, click it in the list. ## Upload lookup tables To upload a lookup table in Observability Platform, first create a spreadsheet in CSV format. Each lookup table must be less than or equal to 10 MB. You can add up to 1,000 lookup tables. To upload lookup tables that exceed 10 MB, contact [Chronosphere Support](/support). To upload lookup tables: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Lookup Tables**. 2. Click **Upload lookup table**. 3. In the **Upload lookup table** page, enter a name and description for your lookup table. Lookup table names must be unique. 4. Click **Upload CSV**, and then select the CSV file you want to upload from your local machine. 5. Click **Save** to upload your lookup table. Your lookup table displays in the list of lookup tables, and can be referenced in log queries using the [`lookup` function](/investigate/querying/query-logs/query-syntax#lookup). ## Edit lookup tables You can replace existing lookup tables by uploading a new lookup table. Replacing a lookup table overwrites the existing table. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Lookup Tables**. 2. Click the lookup table you want to replace. 3. In the **Edit lookup table** page, click **Replace lookup table**. 4. On your local machine, select the CSV file you want to upload. 5. Click **Save** to upload your lookup table. The lookup table is replaced with the one you selected. ## Delete lookup tables To delete lookup tables: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Lookup Tables**. 2. Click the lookup table you want to delete. 3. In the **Edit lookup table** page, click **Delete lookup table**. 4. In the confirmation window, click **Delete** to delete the lookup table. The lookup table is deleted from the list of lookup tables. # Normalize your log data Source: https://docs.chronosphere.io/control/shaping/shape-logs/normalize-logs Learn how to use field mapping in Chronosphere Observability Platform to normalize your log data at ingest time. During the ingestion phase, after parsing your data, Chronosphere Observability Platform applies your preferred field normalization rules to populate predefined keys such as `service`, `severity`, and `timestamp`. ```mermaid actions={false} theme={null} flowchart LR Send["Send data"] subgraph OP["Observability Platform"] Parse[Parse data] Normalize[Normalize data] Control[Control data] Format[Format data] Store[Store data] end Send --> Parse --> Normalize --> Control --> Format --> Store click Parse "/control/shaping/shape-logs/parse-logs" click Control "/control/shaping/shape-logs/control-logs" click Format "/control/shaping/shape-logs/lookup-logs/" classDef stages fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF; class Normalize stages ``` This mapping provides consistency in your data, and optimizes querying in Logs Explorer. After applying these mappings, Observability Platform applies [control rules](/control/shaping/shape-logs/control-logs), and then stores your log data. In addition to these mappings, you can apply custom field mappings to normalize log data, such as normalizing values from several fields that all relate to different environments. See [Examples](#examples) for examples of how you can use custom field mappings. Custom field mappings are applied after log data is ingested, but before Observability Platform stores it. To indicate nesting within a source field, use a dot (`.`). For example, if your logs are in a format like `{"resource": {"labels": {"project_id": "..."}}}`, then enter `resource.labels.project_id`. If your logs already use dot notation, such as `{"resource.labels.project_id": "..."}`, enclose the selector in double quotes and brackets like `["resource.labels.project_id"]`. See [target nested or flat fields with dot notation](#target-nested-or-flat-fields-with-dot-notation) for additional examples. ## View custom field mappings Select from the following methods to view custom field mappings. To view custom field mappings: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. All field mappings display in the **Field normalization** section, including both predefined and custom field mappings. 2. Choose which custom field mappings to view: * To view an individual custom field mapping, click it in the list to open the configuration in the **Edit field mapping** drawer. * To view the code definition for all configured custom field mappings, click the **Code config** tab. To use [Chronoctl](/tooling/chronoctl) to return all custom field mappings, use this command: ```shell theme={null} chronoctl log-ingest-config read ``` [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) in Observability Platform to view a custom field mappings's Terraform representation. To complete this action with the Chronosphere API, use the [`ReadLogIngestConfig`](/tooling/api-info/definition/operations/ReadLogIngestConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create custom field mappings Use one of the following methods to create custom field mappings. Define the custom field mapping definition in Observability Platform, and then [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the **Field normalization** section, click **Create**. 3. In the **Custom field name** field, enter the custom field name to map data to, such as `environment`, `region`, or `userID`. 4. In the **Mapped fields** section, in the **Source field name** field, enter a source field name from your log data to map to the custom field you're creating. Fields are matched in priority order. To add more fields, click **Add field mapping** and then enter another source field. Use a dot (`.`) to indicate nesting within a field. For example, if your logs are in a format like `{"resource": {"labels": {"project_id": "..."}}}`, then enter `resource.labels.project_id`. If your logs already use dot notation, such as `{"resource.labels.project_id"}`, enclose the selector in double quotes and brackets like `["resource.labels.project_id"]`. 5. Optional: In the **Default value** field, enter a default value as the mapping if the custom field contains no values. 6. Optional: Select either **Regex** or **Value map** as the normalization mode. * **Regex**: Enter a regular expression pattern to extract and normalize values, such as for removing extraneous data. For example, use the following regular expression to remove prefixes and suffixes from any environments including `production` in the name: ```shell theme={null} ^.*?(production).*?$ ``` * **Value map**: Enter key:value pairs to map additional metadata within your custom field. For example, map `"warn"` to `"Warning"` and `"err"` to `"Error"` for different severity types. To add more value mappings, click **Add value mapping** and then define the value mapping. 7. Click **Save** to save the custom field mapping definition. 8. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your changes. To use [Chronoctl](/tooling/chronoctl) to create custom field mappings, use this command: ```shell theme={null} chronoctl log-ingest-config create ``` 1. Run the following command to generate a sample configuration you can use as a template: ```shell theme={null} chronoctl log-ingest-config scaffold ``` In the template, `kind: LogIngestConfig` defines an individual log ingest configuration. 2. Use the `custom_field_normalization` object to define the plain text parser to run on your log data. 3. With a completed definition, submit it with: ```shell /FILE_NAME/ theme={null} chronoctl log-ingest-config create -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. To create a custom field mapping with [Terraform](/tooling/infrastructure/terraform): 1. Create a Terraform resource with the `chronosphere_log_ingest_config` resource type, followed by a resource name. 2. Use the `custom_field_normalization` object to define the custom field mapping to run on your log data. 3. Add the definition to a Terraform file. 4. Run this command to create the resource: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`CreateLogIngestConfig`](/tooling/api-info/definition/operations/CreateLogIngestConfig) endpoint. Use the `"custom_field_normalization"` object to define the custom field mapping to run on your log data. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Edit custom field mappings When creating or editing a custom field mapping, you can [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to view code representations of a field mapping. The displayed code also responds to changes you make in the **Visual editor** tab. Select from the following methods to edit custom field mappings. After selecting a field mapping to edit, you can change the field mappings order. Fields are matched in priority order, so an order of `1` has higher precedence over an order of `6`. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the **Field normalization** section, click the custom field mapping you want to edit. 3. In the **Edit field mapping** drawer, change your custom field mapping. 4. To reorder field mappings, drag and drop them to change the order. 5. Click **Save**. Observability Platform saves changes to your custom field mapping. 6. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your changes. To update custom field mappings with [Chronoctl](/tooling/chronoctl), use this command: ```shell theme={null} chronoctl log-ingest-config update ``` 1. Update the custom field mapping definition file. 2. Run the following command to submit the changes: ```shell /FILE_NAME/ theme={null} chronoctl log-ingest-config update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To edit custom field mappings using [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file that updates the resource's existing properties. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` You can also [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to view the custom field mapping's Terraform representation. To complete this action with the Chronosphere API, use the [`UpdateLogIngestConfig`](/tooling/api-info/definition/operations/UpdateLogIngestConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete custom field mappings Select from one of the following methods to delete custom field mappings. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the **Edit field mapping** drawer, in the row of the field mapping you want to delete, click the **** trash can icon. 3. Click **Save** to save your changes. 4. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your changes. Observability Platform deletes your custom field mapping. To delete a custom field mapping with [Chronoctl](/tooling/chronoctl), use this command: ```shell theme={null} chronoctl log-ingest-config delete ``` Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteLogIngestConfig`](/tooling/api-info/definition/operations/DeleteLogIngestConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Target nested or flat fields with dot notation The `source` selectors for custom field mappings support dot notation to read values from fields inside nested structures. How Observability Platform interprets a selector depends on whether you enclose the segments in brackets in double quotes: * An unbracketed dot splits the selector into path segments, which traverses nested fields. A source of `metadata.code` reads a `code` field nested inside a `metadata` object. * A bracket-quoted segment such as `["metadata.code"]` treats the quoted string as a single, literal field name that contains a dot. This reads a flat field named `metadata.code` instead of a nested structure. Consider a custom field mapping that populates a `status_code` target from a source field. To read from a nested field, set the source selector to `metadata.code`: ```json theme={null} { "target": "status_code", "normalization": { "source": [ { "selector": "metadata.code" } ] } } ``` This selector reads the value from the nested `code` field: ```json theme={null} { "metadata": { "code": "500" } } ``` To read from a single flat field whose name contains a dot, enclose the source selector in double quotes and brackets: ```json theme={null} { "target": "status_code", "normalization": { "source": [ { "selector": "[\"metadata.code\"]" } ] } } ``` This selector reads the value from one literal field name: ```json theme={null} { "metadata.code": "500" } ``` ## Examples The following examples show common `custom_field_normalization` patterns. Each object is an element of the `custom_field_normalization` array. ### Standardize and enrich data Normalize values from several fields that all relate to different environments, such as `environment` and `region`. You can then add optional mappings for individual environments, such as mapping `dev` to `development`. ```json theme={null} { "target": "environment", "normalization": { "source": [ { "selector": "env" }, { "selector": "region" } ], "default_value": "production", "value_map": { "dev": "development", "prod": "production", "stg": "staging" } } } ``` ### Consolidate values from multiple fields Map a single `region` field from whichever source field is present, checking `region` first and then `datacenter`. If neither contains a value, apply a default value. ```json theme={null} { "target": "region", "normalization": { "source": [ { "selector": "region" }, { "selector": "datacenter" } ], "default_value": "us-east-1" } } ``` ### Standardize values with a value map Normalize inconsistent HTTP status codes into a smaller set of status classes, and label anything unmatched as `unknown`. ```json theme={null} { "target": "status_class", "normalization": { "source": [ { "selector": "http.status_code" } ], "value_map": { "200": "success", "201": "success", "400": "client_error", "404": "client_error", "500": "server_error", "503": "server_error" }, "default_value": "unknown" } } ``` ### Extract a value with a regular expression Use `sanitize_patterns` to extract a substring from a larger field. Each pattern must contain exactly one capturing group, and Observability Platform uses that group as the result. This example extracts an uppercase log level from a free-form `message` field. ```json theme={null} { "target": "log_level", "normalization": { "source": [ { "selector": "message" } ], "sanitize_patterns": ["^.*level=([A-Z]+).*$"], "default_value": "INFO" } } ``` ### Map a value from a nested field Use dot notation to read a value from a nested structure, such as `user.id` inside `{"user": {"id": "..."}}`. Sources are still matched in priority order. ```json theme={null} { "target": "userID", "normalization": { "source": [ { "selector": "user.id" }, { "selector": "actor.userId" } ] } } ``` ### Map a value from a flat field that contains dots If a source field name already contains dots, bracket-quote the selector so Observability Platform treats it as one literal field name instead of a nested path. ```json theme={null} { "target": "team", "normalization": { "source": [ { "selector": "[\"kubernetes.labels.team\"]" } ], "default_value": "unassigned" } } ``` # Parse your log data Source: https://docs.chronosphere.io/control/shaping/shape-logs/parse-logs Learn how to use parsers in Chronosphere Observability Platform to shape your log data at ingest time. Your log data can be formatted differently depending on which service generates that data. Some data is structured, such as valid JSON. Other data might be somewhat structured, or not have any structure at all. In these instances, you can create parsers to create structured logs from plain text logs, and parse individual fields to extract meaningful data without changing your instrumentation. For example, if your instrumentation emits unstructured plain-text logs, create a [plain text parser](#create-plain-text-parsers) to provide structure by separating the individual fields. If your logs contain JSON or key-value pairs buried contained in a single string field, create a [field parser](#create-field-parsers) to extract the values. The extracted fields can then be queried in [Logs Explorer](/investigate/querying/query-logs) and used in aggregations and transformations. Parsers run after ingestion, but before applying normalization rules and control rules. If you want to additionally parse log data after initial parsing and normalization complete, [create a control rule](/control/shaping/shape-logs/control-logs#create-control-rules) to parse fields. ```mermaid actions={false} theme={null} flowchart LR Send["Send data"] subgraph OP["Observability Platform"] Parse[Parse data] Normalize[Normalize data] Control[Control data] Format[Format data] Store[Store data] end Send --> Parse --> Normalize --> Control --> Format --> Store click Normalize "/control/shaping/shape-logs/normalize-logs" click Control "/control/shaping/shape-logs/control-logs" click Format "/control/shaping/shape-logs/lookup-logs/" classDef stages fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF; class Parse stages ``` ## View parsers Select from the following methods to view and order available parsers. Parsers display in match order. To view parsers: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. All parsers display, including plain-text and field parsers. 2. To view only plain-text parsers or field parsers, from the **View** dropdown, select **Plaintext parsers** or **Field parsers**. 3. To view the code definition for all configured parsers, click the **Code Config** tab. The **Logs parsing and mapping** section includes all defined parsers. To use [Chronoctl](/tooling/chronoctl) to return all parsers, use the `chronoctl log-ingest-config read` command: ```shell theme={null} chronoctl log-ingest-config read ``` [Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) in Observability Platform to view a parser's Terraform representation. To complete this action with the Chronosphere API, use the [`ReadLogIngestConfig`](/tooling/api-info/definition/operations/ReadLogIngestConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create parsers Parsers operate on both [plain-text logs](#create-plain-text-parsers) and [individual fields](#create-field-parsers). All configured parsers are included in a single definition, and evaluated in match order. Observability Platform applies the first parser in the definition that matches a log, and ignores any subsequent parsers. Although you can define parsers in the Observability Platform web interface, you must [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. ### Create plain-text parsers Plain text parsers run against all ingested logs with the `plaintext` mimetype, and apply parsing logic to modify the original record. These parsers affect the entire log. Use field parsers to operate on individual fields within a log. Select either a regular expression or a key-value matching pattern to parse logs. When using a key-value pattern, you can extract any number of key value pairs from a plain text log. Plain text parsers also have an option to keep the original log after applying parsing logic. Plain text parsers apply only when the media type is set to `plaintext` for logs. Regular expressions for the matching type use [RE2 syntax](https://github.com/google/re2/wiki/Syntax) and named capture groups. Each named capture group becomes the key name in the structured log. Observability Platform requires at least one named capture group in the regular expression. Use one of the following methods to create plain text parsers. Define the plain text parser definition in Observability Platform, and then [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the **Plaintext parsers** section, click **Create**. 3. Enter a name for the parser. 4. Select the matching type for the parser: * **Regex**: Use a regular expression to match logs. In the **Template** field, select the regular expression template to apply to your logs. Use a predefined template such as **Nginx**, or specify a **Custom** template. For example, this regular expression is used in subsequent steps for previewing changes: ```text wrap theme={null} ^(?P[^ ]*) [^ ]* (?P[^ ]*) \[(?P To use [Chronoctl](/tooling/chronoctl) to create plain text parsers, use the `chronoctl log-ingest-config create` command: ```shell theme={null} chronoctl log-ingest-config create ``` 1. Run the following command to generate a sample parser configuration you can use as a template: ```shell theme={null} chronoctl log-ingest-config scaffold ``` In the template, `kind: LogIngestConfig` defines an individual parser configuration. 2. Use the `plaintext_parsers` object to define the plain text parser to run on your log data. 3. With a completed definition, submit it with: ```shell /FILE_NAME/ theme={null} chronoctl log-ingest-config create -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. To create a plain text parser with [Terraform](/tooling/infrastructure/terraform): 1. Create a Terraform resource with the `chronosphere_log_ingest_config` resource type, followed by a resource name. 2. Use the `plaintext_parser` object to define the plain text parser to run on your log data. 3. Add the definition to a Terraform file. 4. Run this command to create the resource: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`CreateLogIngestConfig`](/tooling/api-info/definition/operations/CreateLogIngestConfig) endpoint. Use the `plaintext_parsers` object to define the plain text parser to run against your log data. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Create field parsers Field parsers operate on an individual field within a log, and run on all ingested logs, including JSON and plain text logs. Field parsers run after plain text parsers. When defining a field parser, choose which matching pattern to apply and specify the destination of the parsed field. Define the field parser definition in Observability Platform, and then [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the **Field Parsers** section, click **Create**. 3. Enter a source field for the parser to operate on, such as `message`. 4. Enter a destination for the parsed data, which can be one of these locations: * Use the same field name, which overwrites the existing field data with the parsed data. * Specify a new field name to contain the parsed data. * Leave the destination field empty, which populates the parsed data to the top level of your logs. 5. Select the matching type for the parser. * **JSON** matches on valid JSON logs. * **REGEX** uses a regular expression to match logs. In the **Template** field, select the regular expression template to apply to your logs. Use a predefined template such as **Nginx**, or specify a **Custom** template. For example, this regular expression is used in subsequent steps for previewing changes: ```text theme={null} user=(?P\w+) action=(?P\w+) ``` * **Key/value** uses a specified pattern to separate key-value pairs in log data. Enter the following values: * **Pair separator** is the character used to split matching pairs into its key and value. For example, a colon (`:`) if your data is in the format `key:value`. * **Delimiter** is the character used to split the input log data into key-value pairs. For example, a comma (`,`) if your data is in the format `key: value, key: value`. * **Trim set** identifies the leading and trailing characters to trim from the key-value pairs. For example, a space (`" "`) between key-value pairs if your data is in the format `key: value, key: value`. 6. To test the parser, in the **Preview log parser** section, enter a portion of your log payload. For example: ```json theme={null} {"message": "user=john action=login timestamp=12345"} ``` 7. Click **Test parser** to view a preview of the generated output after the parser runs. For example, the following output is generated by the input from the previous step: ```json theme={null} { "message": "user=john action=login timestamp=12345", "parsed_message": { "action": "login", "user": "john" } } ``` 8. Click **Save** to save the parser definition. 9. On the **Logs Ingest** page, click **View code config**. 10. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition. To use [Chronoctl](/tooling/chronoctl) to create field parsers, use the `chronoctl log-ingest-config create` command: ```shell theme={null} chronoctl log-ingest-config create ``` 1. Run the following command to generate a sample parser configuration you can use as a template: ```shell theme={null} chronoctl log-ingest-config scaffold ``` In the template, `kind: LogIngestConfig` defines an individual parser configuration. 2. Use the `field_parsers` object to define the plain text parser to run on your log data. 3. With a completed definition, submit it with: ```shell /FILE_NAME/ theme={null} chronoctl log-ingest-config create -f FILE_NAME ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. To create a plain text parser with [Terraform](/tooling/infrastructure/terraform): 1. Create a Terraform resource with the `chronosphere_log_ingest_config` resource type, followed by a resource name. 2. Use the `field_parser` object to define the field parser to run on your log data. 3. Add the definition to a Terraform file. 4. Run this command to create the resource: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`CreateLogIngestConfig`](/tooling/api-info/definition/operations/CreateLogIngestConfig) endpoint. Use the `field_parsers` object to define the field parser to run against your log data. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Edit parsers When creating or editing a parser, you can [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to view code representations of a parser. The displayed code also responds to changes you make in the **Visual editor** tab. Select from the following methods to edit parsers. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the row of the parser you want to edit, click the three vertical dots icon and then click **Edit parser**. 3. In the **Edit parser** drawer, make changes to your parser, and then click **Save**. Observability Platform saves changes to your parser. 4. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your changes. To update parsers with [Chronoctl](/tooling/chronoctl), use the `log-ingest-config update` command: ```shell theme={null} chronoctl log-ingest-config update ``` 1. Update the parser definition file. 2. Run the following command to submit the changes: ```shell /FILE_NAME/ theme={null} chronoctl log-ingest-config update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To edit parsers using [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file that updates the resource's existing properties. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` You can also [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to view the parser's Terraform representation. To complete this action with the Chronosphere API, use the [`UpdateLogIngestConfig`](/tooling/api-info/definition/operations/UpdateLogIngestConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete parsers Select from the following methods to delete parsers. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Logs Ingest**. 2. In the row of the parser you want to edit, click the three vertical dots icon and then click **Delete parser**. 3. Click the **Code config** tab and [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your changes. Observability Platform deletes your parser. To delete a parser with [Chronoctl](/tooling/chronoctl), use the `chronoctl log-ingest-config delete` command: ```shell theme={null} chronoctl log-ingest-config delete ``` Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteLogIngestConfig`](/tooling/api-info/definition/operations/DeleteLogIngestConfig) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Examples The following example shows how to apply both plain text and field parsers to create structured output from an unstructured log record. Consider the following input log data. The record is a single string that contains multiple key-value pairs in a single `message` field. In this state, querying the data is difficult, and you can't complete aggregations or transformations. ```text theme={null} 2025-09-22 09:15:45.101 2025/09/22 13:15:45 [info] 34567#34567: *9876543210 a new user session was started, client: 203.0.113.55, service: app.example.net, activity: "GET /data/dashboard/overview?appVersion=1.2.3 HTTP/1.1", host: "app.example.net", source: "https://app.example.net/home" ``` By applying a plain text parser and a field parser, you can separate the fixed fields and extract the key-value pairs into a new field that you can operate on in Logs Explorer. First, create a [plain text parser](#create-plain-text-parsers) that uses a regular expression to extract the key-value pairs. Next, add a [field parser](#create-field-parsers) to parse out the values. The following Terraform definition includes both parsers and their definitions: ```terraform expandable Terraform example icon="square-terminal" wrap theme={null} resource "chronosphere_log_ingest_config" "log_ingest_config" { plaintext_parser { name = "metadata" mode = "ENABLED" parser { parser_type = "REGEX" regex_parser { regex = "^(?P\\S+ \\S+)\\s+(?P\\S+ \\S+) \\[(?P\\w+)\\] (?P\\d+)#(?P\\d+): \\*(?P\\d+) (?P.*?), (?P.+)$" } } } field_parser { destination { selector = "metadata" } mode = "ENABLED" parser { parser_type = "KEY_VALUE" key_value_parser { delimiter = "," pair_separator = ":" trim_set = " " } } source { selector = "metadata" } } } ``` After applying both parsers, the generated data is output to a new `metadata` field, and is structured as shown in the following example. The new field includes key-value pairs that can be queried in [Logs Explorer](/investigate/querying/query-logs) and used in aggregations and transformations. ```json theme={null} { "timestamp": "2025-08-22 13:36:18.648", "log_time": "2025/08/22 17:36:18", "log_level": "info", "pid": "34567", "tid": "34567", "connection_id": "9876543210", "message": "a new user session was started", "metadata": { "client": " 203.0.113.55", "service": " app.example.net", "activity": " \"GET /data/dashboard/overview?appVersion=1.2.3\"", "host": " \"app.example.net\"", "source": " \"https://app.example.net/home\"" } } ``` # Shape metric data Source: https://docs.chronosphere.io/control/shaping/shape-metrics Learn how metric shaping rules and downsampling control cardinality, license consumption, and query performance. Managing the cardinality of persisted data can be challenging when determining which metrics to send to Chronosphere Observability Platform. *Metric cardinality* is the number of unique time series produced by a combination of metric names and their associated labels. Cardinality is the total number of these combinations. The more combinations there are, the greater a metric's cardinality is, and the more data Observability Platform persists. This can affect [your license](/administer/limits-licensing/concepts/capacity-licensing#persisted-cardinality). Learn about strategies and tools you can use to [reduce cardinality](/control/shaping/shape-metrics/reduce-cardinality). Observability Platform includes various [*shaping rules*](/control/shaping/shape-metrics/rules) to manipulate the data you send to Observability Platform. Use metric shaping rules to reduce the metrics you store in Observability Platform, optimize query performance, and create aliases for expressions. ## Downsample data [Collectors](/ingest) ingest metrics at specific intervals, based on system configuration. This granularity of metric data can be helpful in diagnostic efforts, but certain issues might not be served by such granularity. If you can diagnose production issues by using a coarser granularity of metric data, *downsampling* the data reduces the amount of data persisted to the Observability Platform database. Downsample incoming data in Observability Platform using these methods: * Change the Chronosphere Collector configuration by [changing the rate](/ingest/metrics-traces/collector/discover/scrape-configuration#global-scrape-configuration) at which the Collector publishes metrics to the server. * Use [mapping rules](/control/shaping/shape-metrics/rules/mapping) to downsample metrics that aren't aggregated. * Use [rollup rules](/control/shaping/shape-metrics/rules/rollup) to downsample aggregated metrics. Over time, persisted data uses significant storage capacity. Observability Platform performs [long-term downsampling](/control/shaping/shape-metrics/downsampling) to control data storage costs while retaining important statistics. Be aware that [late-arriving data](/administer/limits-licensing/limits/metric-limits#late-arriving-metrics) can affect downsampling. # Long-term downsampling Source: https://docs.chronosphere.io/control/shaping/shape-metrics/downsampling Learn how long-term downsampling compacts persisted metrics and its effect on each metric type. After approximately five days, each persisted metric, including both raw and aggregated metrics, undergoes a process called *long-term downsampling*. This process temporally downsamples data into a more compact form, and then deletes the non-downsampled data permanently. To maintain an accurate representation of the data, Chronosphere Observability Platform uses different downsampling methodologies, depending on the metric type. These behaviors are important to note beforehand, since any unexpected results of long-term downsampling will be noticed only approximately five days after ingestion. By default, Observability Platform downsamples long term data at a five-minute granularity, where all data points within each five-minute window compress into a single data point. This five-minute window is termed the *downsample window*. ## Effects on metric types Downsampling effects differ based on the [metric type](/control/shaping/shape-metrics/types). * **Cumulative counter**: [Cumulative counters](/control/shaping/shape-metrics/types#cumulative-counter) downsample by preserving the overall increase (respecting resets) between the start and end of the downsample window. This reduces the temporal granularity by observing only one increase every five minutes, while keeping the running count accurate. * **Delta counters**: [Delta counters](/control/shaping/shape-metrics/types#delta-counter) downsample by preserving the overall increase between the start and end of the downsample window. The `sum_over_time()` should remain the same, but at a reduced granularity, such as every five minutes. * **Gauge**: Downsampling of [gauges](/control/shaping/shape-metrics/types#gauge) differs, depending on how the gauge was ingested or if the gauge originated from a rollup rule. By default, gauges downsample by preserving only the last data point in every downsample window. Any changes to the gauge prior to the end of the downsample window aren't retained. If the gauge is an output of a [`MIN/MAX` aggregation](/control/shaping/shape-metrics/rules/rollup#gauge), the gauge is downsampled by preserving the `MIN/MAX` data point in every downsample window, respectively. Gauges ingested with StatsD downsample using a Largest-Triangle-Three-Buckets (LTTB) downsampling algorithm for consistency with the Graphite query engine. Graphite statistics use a sum downsampling for counters, and LTTB for timers and gauges. * **Histograms**: Cumulative exponential histograms are downsampled by preserving the overall increase (respecting resets) of each bucket count between the start and end of the downsample window. Delta exponential histograms are downsampled by summing the data points in the downsampling window. Due to the reduced temporal granularity, you'll see changes no more frequently than every 5 minutes, while the running bucket counts remain accurate. If the histogram exceeds the 160-bucket limit, Observability Platform decreases its scale until the bucket count is within the limit. Downscaling reduces the histogram's resolution. Classic Prometheus histograms are cumulative counters and have the same downsampling effects. ## Querying downsampled data Querying downsampled data can produce different results, especially between higher and lower resolution data. Some queries can be sensitive to data resolution. Although a query might work well with raw data, it could return no data, or data that doesn't make sense after downsampling. For example, `rate(some_metric[2m])` with a raw resolution of `30s` returns results. If you increase the query time range, the query requests downsampled data. The results are empty, because the query requested multiple data points in the `2m` window, but downsampling reduces available data to one data point every [five minutes](/administer/limits-licensing/licensing#contract-details). To prevent empty graphs when changing the query time range, Observability Platform enforces a minimum range selector on queries that read downsampled data. The minimum is three times the resolution of the downsampled data, which is `15m` at the default `5m` resolution. Observability Platform rewrites any range selector shorter than the minimum to the minimum, and leaves longer range selectors unchanged. A range selector that equals or doubles the resolution is still rewritten, because both remain shorter than three times the resolution. The following table shows how Observability Platform rewrites the previous example at a `5m` resolution: | Original query | Query that runs | Reason | | ------------------------ | ------------------------ | ------------------------------------------------------------- | | `rate(some_metric[2m])` | `rate(some_metric[15m])` | `2m` is shorter than the `15m` minimum. | | `rate(some_metric[5m])` | `rate(some_metric[15m])` | `5m` matches the resolution, but is shorter than the minimum. | | `rate(some_metric[10m])` | `rate(some_metric[15m])` | `10m` is twice the resolution, but shorter than the minimum. | | `rate(some_metric[15m])` | `rate(some_metric[15m])` | `15m` equals the minimum, so no rewrite occurs. | | `rate(some_metric[30m])` | `rate(some_metric[30m])` | `30m` exceeds the minimum, so no rewrite occurs. | Querying lower resolution data might also affect some of the function output, including: * The `increase()` function can return very different numbers after switching to downsampled data. * The `rate()` function smooths out peaks for graphs with many peaks with raw data. * Any `resets()` are dropped when downsampling data. # Understand metric quotas and pools Source: https://docs.chronosphere.io/control/shaping/shape-metrics/quotas Learn about metric quotas and pools, and how to give each pool a specific quota of your total persisted writes license. Metric quotas and pools only apply to capacity pricing. If your organization uses the consumption model, see [Manage consumption](/control/consumption). Metric quotas assign specific percentages of your total persisted writes limit to *pools* of metrics. A pool might translate to teams or other logical groupings within your organization. Use metric quotas to give each pool a specific quota of your total persisted writes license, expressed either as a percentage or a value in Data Points Per Second (DPPS). Chronosphere recommends using the fewest number of pools possible for administrative ease. For most use cases, 10 or fewer pools is adequate. Before adding pools, review concepts about [defining pools](/control/shaping/shape-metrics/quotas/define-pools) to understand how to apply pool allocations, priorities, and thresholds. ## Exceeding limits If a metric, single service, pod, or environment begins to emit a high volume of data that pushes the system over its persisted writes limit, a penalty applies to the pool containing the emitted metric. Pools that haven't exceeded their quota aren't penalized. Individual pool quotas must sum to less than or equal to the [capacity limit](/administer/limits-licensing/concepts/capacity-licensing#capacity-limits). Metrics ingestion for individual pools is guaranteed up to the limit defined for the pool. If your entire Chronosphere Observability Platform tenant is under the capacity limit, an individual pool can exceed its defined quota without incurring a penalty, and Observability Platform allocates the remaining capacity to the **Default Pool**. When a tenant exceeds the overall capacity limit, enforcement rules take effect. How enforcement occurs differs between licenses: * **Persisted and matched writes**: When capacity limits are exceeded for persisted and matched writes, writes are rate limited based on the defined pools and [priority](/control/shaping/shape-metrics/quotas/define-pools#configure-priority). Only pools that exceed their allocation are penalized. In a penalty state, data is dropped in priority order, starting with low-priority data first. * **Persisted cardinality**: When capacity limits are exceeded for persisted cardinality, data is dropped indiscriminately across all pools and priorities until the tenant is under the capacity limit. To control the data that's dropped across pools and priorities for persisted cardinality, you must configure [pool thresholds](/control/shaping/shape-metrics/quotas/define-pools#pool-thresholds). Thresholds proactively limit dropped data to specific pools and priorities before overall capacity is reached. Defining thresholds avoids randomly dropping data, and isolates penalties so you don't impact pools that aren't exceeding their allocation. Configuring pool allocations alone doesn't provide protections for persisted cardinality. Thresholds define limits that prevent series from consuming portions of the overall cardinality budget and inadvertently affecting pools that haven't exceeded their allocation. Defining thresholds means that drops can occur, even if your pool allocation or overall license aren't exceeded. Threshold enforcement takes effect after consumption if the relevant pool exceeds the defined threshold value. This means that if low priority data exceeds the low priority threshold, Observability Platform starts dropping data. The exact point at which enforcement occurs beyond that value can vary. The smaller the threshold is relative to overall capacity, the greater the chance for delay when enforcement takes effect. ## View and manage pools View [existing metric pools](/control/shaping/shape-metrics/quotas/quotas-ui) in Observability Platform on the Metrics Quotas page. If you haven't configured any metric pools, Observability Platform shows a single **Default Pool**. You can also use the [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer) to view the metrics that each pool contains, and filter by priority. You can then [manage pools](/control/shaping/shape-metrics/quotas/manage-pools) to shape and plan your pool sizes, and plan for future usage. # Define a metric pool Source: https://docs.chronosphere.io/control/shaping/shape-metrics/quotas/define-pools Learn about metric pool concepts and how to apply them to help shape defining a metric pool. Before [adding a metric pool](/control/shaping/shape-metrics/quotas/manage-pools#add-a-metric-pool), review these concepts and how you apply them to help shape defining a pool. A metric *pool* groups assigned metrics, which are organized by allocations, priorities, and thresholds: * [Pool allocations](#pool-allocations) define how much of your total persisted writes license you want allocated to each pool. You can define allocations for each of your licenses individually, or have the same allocations across all pools. * [Pool priorities](#configure-priority) let you selectively decide which metrics within a given pool to drop first during a penalty scenario. You can set priorities at a global level (for all pools), or individually. * [Pool thresholds](#pool-thresholds) are a capability that provide more strict control on persisted cardinality on individual pools. Configure thresholds as a proactive measure to strictly enforce dropping data, even when the overall capacity limit hasn't been exceeded. * [Match rules](#match-rules) are the set of rules that determine which metrics belong to the pool. A metric has to match only one of the rules to belong to the pool. ## License impact Any applied quota configuration displays in the [Metrics Quotas](/control/shaping/shape-metrics/quotas/quotas-ui) page, which shows how the pool's traffic interacts with the pool's quota, without penalizing that pool if the system goes over its limit. * To understand matched writes quotas and how they applies to individual pools, see the [Matched Writes Quotas dashboard](/observe/dashboards/managed-dashboards#matched-writes-quotas). * To understand persisted cardinality and how it applies to individual pools, see the [Persisted Cardinality Quotas dashboard](/observe/dashboards/managed-dashboards#persisted-cardinality-quotas). You can also access these dashboards from the [License Overview](/observe/dashboards/managed-dashboards#licensing-dashboards) by clicking the three dots icon in the dashboard panel relating to the license type, and then clicking the dashboard. When defining pool allocations, you can specify `fixed_values` for a license, you can specify any of the supported license types, which include persisted writes, persisted cardinality, and matched writes. When specifying `priority_thresholds`, you can specify only persisted cardinality license dimensions, defined by `PERSISTED_CARDINALITY_STANDARD` and `PERSISTED_CARDINALITY_HISTOGRAM`. See the [CreateResourcePools endpoint](/tooling/api-info/definition/operations/CreateResourcePools) for more information. ## Pool allocations You can define pool allocations either as a percentage `(percent_of_license)` that applies to all pools for all licenses, or as a fixed value `(fixed_values)` in data points per second (DPPS) for individual licenses. Any remaining capacity within each license is allocated to the default pool, after subtracting the sum of allocations across pools for that license. The default pool allocation is calculated based on remaining capacity. To calculate the data allocation for the default pool, subtract the sum of allocations across pools for persisted writes, minus the overall [capacity limit](/administer/limits-licensing/concepts/capacity-licensing#capacity-limits): ```text theme={null} Capacity limit - allocations across pools = default pool allocation ``` The remainder is assigned to the default pool. You can specify any combination of `percent_of_license` and `fixed_values` for each license dimension. However, all pools within a license dimension must use the same units. For example, if matched writes uses `percent_of_license`, all pools must use that unit for matched writes. Similarly, if persisted writes uses `fixed_values`, all pools must use fixed value for persisted writes. * `percent_of_license`: Specify the percentage of the license to allocate to a pool. This value applies to any license dimensions without `fixed_values` defined. For example, consider that you allocate 10% of your persisted cardinality license to a pool, and your total persisted cardinality license is `5.0M` (million). The allocation for this pool is `5.0M` of persisted cardinality (.10\*50), so any consumption beyond that value causes metrics to be dropped. * `fixed_values`: Specify a fixed value for a license dimension in DPPS. You can set a single fixed value per license dimension. Any `fixed_values` take precedence over `percent_of_license` for a given license. If you set any `fixed_values`, you can specify allocations for both matched writes license and persisted writes. These allocations are available for both standard and histogram metrics. See the [CreateResourcePools endpoint](/tooling/api-info/definition/operations/CreateResourcePools) for more information. The sum of fixed values across all defined pools must be less than or equal to the total allotted capacity, defined by the [capacity limit](/administer/limits-licensing/concepts/capacity-licensing#capacity-limits). If your organization exceeds the capacity limit, where the sum of fixed values exceeds total capacity, a penalty is applied to all pools proportional to fixed allocations. A validation in Terraform penalizes any pool that exceeds its allotted quota. In this penalty state, the default pool receives no allocation, and other pools are adjusted down proportionally so that the sum of fixed values is equal to the capacity limit. ## Configure priority If you configured metrics quotas and your system exceeds its license limit, Observability Platform drops metrics from pools that exceed their respective quotas until all pools meet their quotas. Observability Platform penalizes only pools that exceed their persisted writes quota. To more selectively decide which metrics within a given pool to drop first during a penalty scenario, specify priorities for each pool: * **High**: Metrics dropped last. * **Low**: Metrics dropped first. * **Default**: Metrics dropped after low priority metrics, but before high priority metrics. For persisted writes and matched writes, Observability Platform uses these priorities to determine the order of drops if your organization exceeds their capacity limit. These priorities are also used in conjunction with [setting thresholds](#pool-thresholds). Chronosphere recommends creating a pool for each team in your organization so they can manage their own budgets. For example, the Ordering Team who's responsible for the ordering service can manage their own budget with a pool that's specific to their team. If the ordering service is allotted 20% of the overall budget, the Ordering Team can configure priorities within that pool and follow the [best practices](/control/shaping/shape-metrics/quotas/manage-pools#best-practices) to proactively manage their budget. ### Configure global priority You can change global pool quota configurations by metric label. Any changes to quota configuration labels require updates to all pools. 1. On the **Metrics Quotas** page, click **Configure Quotas**. 2. Click **Edit Global Settings**. 3. In the **Edit Global Pool Settings** dialog, select a label from the **Quota Configuration Label** dropdown. This label is the label key that defines which keys can be used to create a pool. 4. Select **Configure Globally** to apply the pool filtering globally, and complete the following fields: * **Prioritization label**: Select a label to change its priority. * **High priority values**: Add a label value, such as `production*` to ensure metrics with that label value are retained. * **Low priority values**: Add a label value, such as `test*` to drop metrics of lower importance first. 5. Click **Done** when finished. 6. Click the **Code Config** tab. 7. Click ** Copy** to copy the file, or ** Download** to download the file to your computer. 8. Add the definition to a Terraform file, or create a new Terraform file. 9. Run this command to apply the resource: ```shell theme={null} terraform apply ``` ### Configure priority per pool You can configure priority for each pool instead of [configuring priority globally](#configure-global-priority). Complete the following steps to configure priority for each pool individually. 1. On the **Metrics Quotas** page, click **Configure Quotas**. 2. Click **Edit Global Settings**. 3. In the **Edit Global Pool Settings** dialog, select a label from the **Quota Configuration Label** dropdown. This label is the label key that defines which keys can be used to create a pool. 4. Select **Configure per pool** to set priority independently for each pool. 5. Click **Done** when finished. 6. [Edit each pool](/control/shaping/shape-metrics/quotas/manage-pools#edit-a-pool) to set priorities. ## Pool thresholds After configuring [pool priorities](#configure-priority), administrative users should configure thresholds on individual pools to manage persisted cardinality. You must configure pool thresholds to set strict limits for when to drop new consumption of the persisted cardinality license for a pool. Thresholds let you strictly enforce certain pools at configured values, even if the overall capacity limit wasn't exceeded. By proactively limiting series in strictly enforced pools, thresholds prevents those series from consuming portions of the overall cardinality budget and inadvertently affecting pools that haven't exceeded their allocation. If you don't configure pool thresholds, when a pool exceeds its defined allocation, drops occur randomly across all pools. You must configure pool thresholds to enforce limits on individual pools. Configuring pool allocations alone doesn't provide protections for persisted cardinality. Administrative users can configure the following thresholds to take a more proactive approach to budget optimization, enabling them to implement incremental steps to prevent an overage before it occurs. ### All priorities threshold To help solve the "noisy neighbor" problem and isolate the impact of changes to individual teams, strictly enforce the all priorities (`all_priorities.percent_of_pool_allocation`) threshold. This threshold stops accepting data of any priority if consumption for the pool exceeds the specified threshold value. This threshold limits churn in a pool from exceeding the defined threshold and inadvertently affecting other pools. If you don't set the threshold for all priorities, an overage could result in drops across other pools. Medium and high priority data can continue to consume capacity until the entire pool allocation is exhausted. Setting the threshold for all priorities helps ensure that a pool's consumption doesn't impact other high priority metrics for other pools. When configuring the all priorities threshold, consider the following best practices: * The value of this threshold should be greater than or equal to the pool allocation. For example, if the pool allocation is 50%, then the threshold should be 50% or greater. If the threshold value is less than the pool allocation, consider changing the pool allocation instead. Setting the threshold to a value that exceeds 100% of the pool allocation can cause multiple pools to exceed their allocation, which can cause your system to hit the [cardinality limit](/administer/limits-licensing/concepts/capacity-licensing#how-persisted-cardinality-limits-work) and arbitrarily drop data. * Use a percentage for this value rather than a fixed value in DPPS, especially if you want the threshold value to match the pool allocation. If the threshold is a fixed value, then you must update that value any time the pool allocation or license capacity changes. * You can set this threshold to a value that exceeds 100% of the pool allocation, which provides a buffer before the threshold limit causes the pool to drop data. This configuration accommodates occasional spikes, or situations when you're migrating data from one pool to another over a longer period. ### Low and medium priority thresholds To proactively limit churn for low and medium-priority series, configure low and medium priority thresholds to ensure there's room in your license for high-priority series. The low priority threshold (`low_priority`) stops accepting low-priority data only if the data exceeds the threshold value. This threshold limits churn in low-priority series that exceed the threshold. The low and medium priority threshold (`default_and_low_priority`) stops accepting low and medium priority data if low- and medium-priority data combine to exceed the threshold value. This threshold limits churn in low- and medium-priority series that exceed the threshold. ### Configure pool thresholds Configuring pool thresholds is supported only in Terraform and the [`CreateResourcePools`](/tooling/api-info/definition/operations/CreateResourcePools) endpoint of the [Chronosphere API](/tooling/api-info). To determine the threshold values for each pool, use the [Persisted Cardinality Quotas dashboard](/observe/dashboards/managed-dashboards#persisted-cardinality-quotas) to identify usage trends. The data in this dashboard can help inform recommended thresholds for low, medium, and high priority series in each pool. For example, * If the control team consistently uses approximately 95% of its pool, and you want to prevent an overage from resulting in drops in other pools, set a strict threshold (`all_priorities`) to 100%. * If the control team contains data for your most important pool, and you'd rather drop data from all other pools except this one, set strict thresholds for all other pools except the control team pool. * If the control team often experiments with new series in its development and staging environments, set proactive thresholds to 10% for low priority and 20% for combined low and medium priority data to preserve space for high-priority series. Complete the following steps to set pool thresholds. See the [example](#pool-thresholds-example) for how to configure thresholds within a pool. 1. Add the `priority_thresholds` object to your existing metric pools definition with either Terraform or the [`CreateResourcePools`](/tooling/api-info/definition/operations/CreateResourcePools) endpoint. 2. Define the license you want the threshold to operate on. Thresholds support these licenses: * `PERSISTED_CARDINALITY_STANDARD`: Refers to the [standard metric license](/administer/limits-licensing/concepts/capacity-licensing#standard-metrics-license), which measures the current consumption rates across persisted writes, matched writes, and persisted cardinality license dimensions measured against the capacity limit. * `PERSISTED_CARDINALITY_HISTOGRAM`: Refers to the [histogram metrics license](/administer/limits-licensing/concepts/capacity-licensing#histogram-metrics-license), which measures the current consumption rate across all histogram metrics license dimensions measured against the capacity limit. 3. Define the thresholds you want to configure, which can be one of the following values: * `all_priorities`: Stop accepting any data (low, medium, and high) at the specified threshold if consumption for the pool exceeds threshold value. * `default_and_low_priority`: Stop accepting low and medium priority data at the specified threshold if low and medium priority data combined exceed threshold value. * `low_priority`: Stop accepting only low priority data at the specified threshold if low priority data exceeds threshold value. 4. Save and apply your metric pools definition. After making changes, use the [Persisted Cardinality Quotas dashboard](/observe/dashboards/managed-dashboards#persisted-cardinality-quotas) to track which pools are approaching or exceeding defined thresholds, identify where drops are occurring, and view which priority levels are affected. After updating definitions for priorities or pools, only new inbound time series adhere to the new rules immediately. Any existing, inactive series that were already attributed to a changed pool might continue to count towards your cardinality limit until they naturally expire in the 150 minute rolling window. This means that it might take 150 minutes for cardinality per pool and per priority to accurately reflect counts. #### Pool thresholds example In the following example, priority thresholds are set for `PERSISTED_CARDINALITY_HISTOGRAM` for `all_priorities` and `low_priorities`: ```terraform expandable Terraform example icon="square-terminal" theme={null} pool { name = "Control Services" allocation { percent_of_license = 16 priority_thresholds { license = "PERSISTED_CARDINALITY_STANDARD" all_priorities { percent_of_pool_allocation = 100 } default_and_low_priority { percent_of_pool_allocation = 50 } low_priority { percent_of_pool_allocation = 25 } } priority_thresholds { license = "PERSISTED_CARDINALITY_HISTOGRAM" all_priorities { percent_of_pool_allocation = 100 } low_priority { percent_of_pool_allocation = 25 } } } match_rules = ["service:{${join(",", control_services)}}"] priorities { high_priority_match_rules = ["cluster:production*"] low_priority_match_rules = ["cluster:test*"] } } ``` ## Match rules Match rules are filters that define which metrics are high or low priority. Any metric that matches at least one filter is considered high or low priority, depending on the defined priority. High priority match rules take precedence over low priority ones. If a metric matches both a high and low priority rule, it's considered a high priority metric. When the license limit is exceeded, high priority metrics are dropped last, and low priority metrics are dropped first. This behavior applies to only persisted writes and matched writes, but doesn't apply to persisted cardinality. Match rules support [glob syntax](/investigate/querying/glob-syntax). See the `match_rules` defined in the [pool thresholds example](#pool-thresholds-example) and the [Terraform pool example](/control/shaping/shape-metrics/quotas/manage-pools#terraform-pool-example) to understand the rule syntax. # Manage metric pools Source: https://docs.chronosphere.io/control/shaping/shape-metrics/quotas/manage-pools Add, edit, and preview metric pool quota allocations on the Metrics Quotas page in Chronosphere Observability Platform. On the [Metrics Quotas](/control/shaping/shape-metrics/quotas/quotas-ui) page in Chronosphere Observability Platform, review current quota allocations and preview potential changes by pool. Use this feature to plan pool sizes and future usage. To get started with pools, learn about how to [define a pool](/control/shaping/shape-metrics/quotas/define-pools). ## Add a metric pool Select from the following methods to add a metric pool. Changes made in the web interface are previews that you apply using the generated configuration. You can define up to 20 custom metric pools. Although actual management of pools is handled using [Terraform](/tooling/infrastructure/terraform), this interface helps you understand what changes to make to reduce guesswork and repeated updates to your system. To create quota pools, you must have administrative privileges: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Metrics Quotas**. 2. Click **Configure Quotas**. 3. Click **Add Pool**. If **Add Pool** is unavailable, expand **Global Settings**, click **Edit Global Settings**, and select a **Quota Configuration Label**. 4. The following fields appear in the drawer. Update editable fields to modify your pool configuration: * **Pool name:** (Editable) Change the pool name. * **Data matching:** The values the selected pool uses to match data. * **Quota configuration label:** The label matching this pool. * **Data matching values:** (Editable) The specific values for the label, which match this pool. Add a value and press `Enter` or Space to add it to the list. The value supports [glob syntax](/investigate/querying/glob-syntax). * **Observed label consumption:** Displays estimated Standard Metrics consumption for the matching label values, broken down by label value. The estimate isn't restricted to data currently assigned to the pool. * **Quota allocation:** (Editable) Set a quota percentage for the selected pool. The drawer displays the equivalent persisted writes Standard Metrics value in data points per second (DPPS). To enter a DPPS value, use the **DPPS** mode in the **Pools** table. * **Prioritization:** (Conditionally editable) Add the **Priority Label** and high or low priority values. Pools using a global priority setting can't change their priorities for an individual pool. 5. In the **Quota Allocation** chart, select or clear pool names in the legend to include or remove their segments from the **Allocation** and **Current Consumption** stacked bars. 6. Click **Done** after completing your changes. 7. Click the **Code config** tab. 8. Click ** Copy** to copy the file, or ** Download** to download the file to your computer. 9. Review the generated configuration. Percentage changes apply to licenses that don't have fixed-value overrides. Changes entered in **DPPS** mode update only the `PERSISTED_WRITES_STANDARD` fixed value. The generated configuration preserves existing fixed values and thresholds for other licenses. 10. Add the definition to a Terraform file, or create a new Terraform file. 11. Run this command to apply the resource: ```shell theme={null} terraform apply ``` To create a pool using [Chronoctl](/tooling/chronoctl): 1. Run the following command to generate a sample pool configuration you can use as a template: ```shell theme={null} chronoctl resource-pools scaffold ``` In the template, `kind: ResourcePools` defines the complete pool configuration. The `ResourcePools` resource is a singleton that contains the default pool and all named pools. If a `ResourcePools` resource already exists, follow the [Chronoctl edit procedure](#edit-a-pool) instead of creating another resource. 2. With a completed definition, submit it with: ```shell theme={null} chronoctl resource-pools create -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. To create a pool with [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file and add the definition by using the `chronosphere_resource_pools_config` type, followed by a name in a resource declaration. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` See the [Terraform pool example](#terraform-pool-example) for a completed pool resource. To complete this action with the Chronosphere API, use the [`CreateResourcePools`](/tooling/api-info/definition/operations/CreateResourcePools) endpoint. The endpoint creates the complete `ResourcePools` singleton. If the resource already exists, use the [`UpdateResourcePools`](/tooling/api-info/definition/operations/UpdateResourcePools) endpoint to add a named pool. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} theme={null} theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Chronoctl pool example The following code is an example of a Chronoctl resource used to create quotas and priorities. The example creates two pools: one for `team_a` and one for `team_b`. Each pool defines license values for persisted writes and matched writes, along with [thresholds](/control/shaping/shape-metrics/quotas/define-pools#pool-thresholds) to manage persisted cardinality. ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: ResourcePools spec: pools: - name: team_a allocation: fixed_values: - license: PERSISTED_WRITES_STANDARD value: "70000" - license: MATCHED_WRITES_STANDARD value: "30000" priority_thresholds: - license: PERSISTED_CARDINALITY_STANDARD all_priorities: fixed_value: "6000000" low_priority: fixed_value: "2000000" filters: - name: team value_glob: 'a' priorities: high_priority_filters: - name: env value_glob: '*prod*' low_priority_filters: - name: env value_glob: '*dev*' - name: team_b allocation: fixed_values: - license: PERSISTED_WRITES_STANDARD value: "16000" - license: MATCHED_WRITES_STANDARD value: "15000" priority_thresholds: - license: PERSISTED_CARDINALITY_STANDARD all_priorities: fixed_value: "1750000" low_priority: fixed_value: "750000" filters: - name: team value_glob: 'b' priorities: high_priority_filters: - name: env value_glob: '*prod*' low_priority_filters: - name: env value_glob: '*dev*' ``` ### Terraform pool example The following code is an example of a Terraform file used to create pools and priorities. This code is an example and shouldn't be used directly. This example uses the following variables in place of a specific name: * *`MY_RESOURCE`* is a resource in your system. * *`MY_SERVICE`* is the name of a service in your system. ```terraform expandable Terraform example icon="square-terminal" theme={null} resource "MY_RESOURCE_resource_pools_config" "resource_pools" { default_pool { priorities { high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"] low_priority_match_rules = ["MY_RESOURCE_k8s_cluster:rc*"] } } pool { name = "Tracing Services" allocation { # Applies to each license that doesn't have a fixed-value override. percent_of_license = 10 } match_rules = ["service:{spanhandler,traceingester}"] priorities { # Optional. Match rules are filters that define which metrics are high or # low priority. Any metric that matches at least one filter is considered # high or low priority, depending on the defined priority. High priority # match rules take precedence over low priority ones. If a metric # matches both a high and low priority rule, it's considered a high # priority metric. When the license limit is exceeded, high priority # metrics are dropped last, and low priority metrics are dropped first. # This behavior applies to only persisted writes and matched writes, but # doesn't apply to persisted cardinality. high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"] low_priority_match_rules = ["MY_RESOURCE_k8s_cluster:rc*"] } } pool { name = "Collector service" allocation { percent_of_license = 8 # Optional. For supported licenses, defines thresholds with strict limits # for when to drop new consumption of the license for a pool. Currently, only # `PERSISTED_CARDINALITY_STANDARD` and `PERSISTED_CARDINALITY_HISTOGRAM` are # supported. priority_thresholds { license = "PERSISTED_CARDINALITY_STANDARD" # Threshold limit that defines when to drop new metrics in the pool. This # threshold applies to all priorities of metrics: high, default, and low. # This field must be set, and have a value equal to or greater than other # priority fields. all_priorities { percent_of_pool_allocation = 100 } } priority_thresholds { license = "PERSISTED_CARDINALITY_HISTOGRAM" all_priorities { percent_of_pool_allocation = 100 } } } match_rules = ["service:{${join(",", local.collector_services)}}"] priorities { high_priority_match_rules = ["chronosphere_k8s_cluster:prod*"] low_priority_match_rules = ["chronosphere_k8s_cluster:rc*"] } } pool { name = "MY_SERVICE Services" allocation { percent_of_license = 25 } match_rules = ["service:MY_SERVICE*"] priorities { high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"] low_priority_match_rules = ["MY_RESOURCE_k8s_cluster:rc*"] } } pool { name = "Gateway Services" allocation { percent_of_license = 4 } match_rules = ["service:gateway*"] priorities { high_priority_match_rules = ["MY_RESOURCE_k8s_cluster:production*"] low_priority_match_rules = ["MY_RESOURCE_k8s_cluster:rc*"] } } } ``` ## Edit a pool Select from the following methods to edit pools. You can also [configure global priorities](/control/shaping/shape-metrics/quotas/define-pools#configure-global-priority) to change global pool quota configurations by metric label. The visual editor requires one quota configuration label across all pools and one prioritization label within each pool. Edit configurations that use multiple matching or prioritization labels directly in Terraform. To edit an existing pool: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Metrics Quotas**. 2. Click **Configure Quotas**. 3. Click any row in the **Pools** table to open the **Edit Pool** drawer. The **Edit Pool** drawer contains information specific to the selected pool. These fields match the **Add Pool** screen, and some values can be edited. 4. Make any necessary changes, and then click **Done** after completing your changes. 5. Click the **Code config** tab. 6. Click ** Copy** to copy the file, or ** Download** to download the file to your computer. 7. Review the generated configuration to confirm that existing fixed values and priority thresholds are preserved. 8. Add the definition to a Terraform file, or create a new Terraform file. 9. Run this command to apply the resource: ```shell theme={null} terraform apply ``` To edit a pool using [Chronoctl](/tooling/chronoctl): 1. View your existing metrics pools YAML definition with the `chronoctl resource-pools read` command: ```shell theme={null} chronoctl resource-pools read ``` 2. Copy the YAML definition and save it to a new file ending in `.yaml`. 3. Modify the YAML definition properties and apply the changes: ```shell theme={null} chronoctl resource-pools update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. To edit a pool using [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file that updates the resource's existing properties. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` You can also use the Code Config tool to view and edit the resource pool Terraform representation: 1. Click the **Code Config** tab. 2. Make changes to the resource pool definition. 3. Click ** Copy** to copy the file, or ** Download** to download the file to your computer. 4. Add the definition to a Terraform file, or create a new Terraform file. 5. Run this command to apply the resource: ```shell theme={null} theme={null} theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`UpdateResourcePools`](/tooling/api-info/definition/operations/UpdateResourcePools) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} theme={null} theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Understanding pool usage The **Pools** section of the **Configure Quotas** page describes what pools you have, and how they're configured. This includes: * **Pool name:** The display name of the pool. * **Data matching:** The label values the pool matches. * **Allocation:** The percentage or DPPS of total traffic guaranteed to the pool before it might be penalized. * **Consumption:** The percentage or DPPS of total traffic the pool is consuming over the selected [time range](/navigate/time-ranges). Allocation and consumption DPPS values describe [Standard Metrics License](/administer/limits-licensing/concepts/capacity-licensing#standard-metrics-license) allocation and consumption. Histogram Metrics License allocation and consumption aren't included in the pool's reported DPPS. #### Quota allocations and consumption The **Quota Allocations vs Current Consumption** graph compares two stacked bars. The **Allocation** bar segments the current quota allocation by pool. The **Current Consumption** bar segments current consumption by pool. Select pool names in the legend to filter both bars. #### Quota consumption by pools (per second) The **Quota Consumption by pools** graph compares each pool's displayed consumption with its current quota limit. Changing an allocation adds the **New Quota Limit** series. #### Understand quota consumption trends The **Quota Consumption** graph plots each pool's consumption over time. Point to a data point to display its value. Drag across the graph to focus on a time period. #### Preview quota allocations The **Pools** include a group of text fields corresponding to each created pool. These boxes contain values with the assigned percentage (**%**) or DPPS for each pool. Use these to set your general pool allocations. To preview a new quota allocation, change a number in the box for the pool to be updated. Click outside the boxes to update the total. Quota settings must meet the following criteria: * In the web interface, changing a named pool's quota adjusts the default pool so that all allocations total 100%. * The **Add Pool** and **Edit Pool** drawers support values greater than or equal to 0.01%. Allocation fields in the **Pools** table support positive values with up to three decimal places. * The API supports values from 0% through 100% with up to three decimal places. If the default pool allocation is omitted, named pool allocations can total less than 100%, and the default pool receives the remainder. Changing an assigned quota displays a third bar in the **Quota consumption by pools** chart. Use the new bar to determine if new quota assignments meet the needs of each of your pools. If one pool is consistently over quota and the other pools aren't, use the preview to adjust assigned quotas to better meet the needs of each pool. Click the ** Reset quotas** icon to restore existing allocation values. This action doesn't undo changes to names, data matching, priorities, added pools, or removed pools. ## Delete a pool Select from the following methods to delete pools. To delete an existing pool: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Metrics Quotas**. 2. Click **Configure Quotas**. 3. Click a non-default pool's row in the **Pools** table to open the **Edit Pool** drawer. 4. At the bottom of the **Edit Pool** drawer, click **Remove Pool**. The pool is removed from the preview and the resource definition in the **Code config** tab. 5. Click the **Code config** tab. 6. Click ** Copy** to copy the file, or ** Download** to download the file to your computer. 7. Update your Terraform file with the generated definition. 8. Run this command to apply the change: ```shell theme={null} terraform apply ``` To delete a pool using [Chronoctl](/tooling/chronoctl): 1. View your existing metrics pools YAML definition with the `chronoctl resource-pools read` command: ```shell theme={null} chronoctl resource-pools read ``` 2. Copy the YAML definition and save it to a new file ending in `.yaml`. 3. Remove the pool you want to delete from the YAML definition and save the updated file. 4. Apply the changes: ```shell theme={null} chronoctl resource-pools update -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. To delete the *entire* `ResourcePools` definition, use the `chronoctl resource-pools delete` command: ```shell theme={null} chronoctl resource-pools delete ``` This command deletes the *entire* `ResourcePools` definition. To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} theme={null} theme={null} terraform apply ``` You can also use the Code Config tool to view and edit the resource pool Terraform representation: 1. Click the **Code Config** tab. 2. Make changes to the resource pool definition. 3. Click ** Copy** to copy the file, or ** Download** to download the file to your computer. 4. Add the definition to a Terraform file, or create a new Terraform file. 5. Run this command to apply the resource: ```shell theme={null} theme={null} theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteResourcePools`](/tooling/api-info/definition/operations/DeleteResourcePools) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} theme={null} theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. This action deletes the *entire* `ResourcePools` definition. ## Best practices To keep penalty behavior and cost accounting transparent and predictable, pools should be hard partitions of your system, with no one time series matching more than one pool. The following processes help ensure pools have the correct data: * Chronosphere recommends selecting a single usage tag as the pool assignment mechanism. Picking a single tag reduces the possibility where one pool matches `serviceX`, and a second pool matches `environmentY`, where time series might match either or both definitions. * Use exact match values for the selected label. The API rejects an exact label value that's already assigned to another pool. * Observability Platform uses match ordering for valid glob patterns that overlap. If a time series matches more than one pool, it becomes part of the first matching pool in the list. * If you see a pool that doesn't match the expected penalty behavior, open the pool in the profiler and compare it with the Terraform configuration file. A match rule value might be incorrect. ## Create an alert for a pool in penalty When a pool is in a penalty state, it might drop metrics to reduce usage. For higher priority pools, this can result in the loss of important data. To reduce or prevent data loss, create a [monitor](/investigate/alerts/monitors) to alert on pool usage and [send notifications](/investigate/alerts/notifications) to the appropriate team. 1. [Create a notifier](/investigate/alerts/notifications/notifiers) and align with your internal alerting policies to route the alerts to the right team. 2. Create a [notification policy](/investigate/alerts/notifications/policies) that connects the notifier to a monitor. 3. [Create a monitor](/investigate/alerts/monitors#create-a-monitor). 4. In the monitor query, add the query to return each pool's percent utilization. For example, the following query calculates persisted writes standard metrics utilization and excludes pools with zero allocation: ```text wrap theme={null} ( 100 * sum by (pool_name) ( chrono_metrics_persisted_writes_license_dpps_consumed{datapoint_type="standard"} ) / sum by (pool_name) ( chrono_metrics_persisted_writes_license_dpps_capacity{datapoint_type="standard"} ) ) and on (pool_name) ( sum by (pool_name) ( chrono_metrics_persisted_writes_license_dpps_capacity{datapoint_type="standard"} ) > 0 ) ``` 5. In the **Signals** section, select **Per time series (many alerts)** as the [signal](/investigate/alerts/notifications/signals#per-time-series). 6. Define the warning condition as greater than 90 and set the sustain period to 5 minutes. 7. Click **Save** to save the monitor. # View allocated pools Source: https://docs.chronosphere.io/control/shaping/shape-metrics/quotas/quotas-ui View the persisted writes quota allocated to and consumed by each metrics pool to find pools at risk of being limited. The **Metrics Quotas** page displays existing metrics pools, the persisted writes quota allocated to each pool, and the amount of persisted writes quota consumed by each pool. Teams can use this information to: * Understand which pools are consuming quota and how much. * Identify which pools are at risk of being limited. * Determine how each pool has prioritized its traffic to ensure that only lower priority data drops if the system exceeds its limit while a pool is over its quota. Select from the following methods to view your metrics pools. 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Metrics Quotas**. 2. Use the **Sort by** menu to change the display order. A summary of **All Pools** always display at the top of the list. 3. Use the **Search** box to find a specific pool. 4. Select a **Time Range** to display. 5. Select a pool name to view data specific to that pool. The **Default Pool** is the pool containing all otherwise unassigned metrics. Hold the pointer over any sparkline chart to display a dialog with data specific to that day and time. Click the dialog to pin it. To list your existing metrics pools using [Chronoctl](/tooling/chronoctl), use the `chronoctl resource-pools read` command: ```shell theme={null} chronoctl resource-pools read ``` To complete this action with the Chronosphere API, use the [`ReadResourcePools`](/tooling/api-info/definition/operations/ReadResourcePools) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## All Pools When viewing **All Pools**, you can see system-wide persisted writes quota consumption and the pools consuming the highest percentage of their assigned quotas. The upper portion of this page displays persisted writes statistics across all pools: * **Limit** total for persisted writes, in data points per second (DPPS). * **Average** persisted writes in DPPS. * **Current** percentage of usage versus your limit. * **Total over time** is a graph of the change over time in the overall consumption of quota, as a percentage of total quota. The **All Pools** page includes additional charts and statistics: * **Overall limit and consumption by pool** is a stacked-area sparkline chart of individual pools and how much quota they're consuming. Use this chart to compare pools and better understand which of them are using the most resources. Hold the pointer over the chart to view the number of data points contributed by each pool at that time. * **Quota consumption by pool** graphs detail the assigned limits and current consumption of that limit for each pool. If one of your pools exceeds its limit, but the overall quota consumption is still 100%, the pool exceeding the limit continues to collect data normally. Hold the pointer over a graph to see incoming data points and the quota limit. When the system exceeds the persisted writes limit, Chronosphere Observability Platform enforces the limit by dropping data from pools over their quota. * **License overage state** shows the volume of metrics dropped for each pool that's in a penalty state. Any pool that exceeds the persisted writes license capacity displays in this chart. ## Select a pool Click a pool name to view data specific to that pool. The pool's page includes the percentage of persisted writes the selected pool consumes, and overall quota consumption against the total persisted writes limit. Each pool page also includes the following details about the pool's quota usage: * **Top persisted writes by service** can help determine which services configured in this pool generate the most writes. * **Persisted writes trends by service** displays the current percentage of total quota this pool is using, and a graph tracking the usage of the time range. * **Quota consumption** tracks the consumption of quota in DPPS, against the limit. * **Traffic priority**: Displays traffic priority in a box chart that shows which priorities have the most traffic, based on [priority definitions](/control/shaping/shape-metrics/quotas/manage-pools). In addition to any named metric pools you define, all instances have a **Default pool**. You can set the quota for the default pool, but not the name or match criteria. The default pool matches any metric that isn't matched by another pool. New metrics that don't match defined pools get assigned to the **Default pool**. ### License overage dropped metrics If the aggregate usage exceeds the [Persisted Writes Capacity limit](/administer/limits-licensing/concepts/capacity-licensing#persisted-writes), the pools exceeding their defined quotas are penalized. In this penalty state, Observability Platform can start dropping data points. During a penalty period, the **License overage dropped metrics** pane displays the total volume of dropped metrics for each pool, and the associated priority of high, medium, and low. Use these graphs to identify when Observability Platform started dropping metrics. You can then create [drop rules](/control/shaping/shape-metrics/rules/drop-rules) to drop incoming metrics and create [aggregation rules](/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules) to reduce persisted writes. ### Quota Configuration Values This table lists all the label **Values** matching the pool configuration. and how much DPPS each contributes to the pool quota. In the **Default pool**, new metrics have the **New** indicator. ### Quotas for the Histogram Metrics License Observability Platform offers limited support of metric quotas for metrics that consume [Histogram Metrics License](/administer/limits-licensing/concepts/capacity-licensing#histogram-metrics-license) capacity. The percentage quota defined for a pool applies to both Standard and Histogram Metrics. For example, when you allocate a 30% quota to a pool, the pool's quota is 30% of your Standard Metrics License's persisted writes and 30% of your Histogram Metrics License's persisted writes. However, the Metrics Quotas UI doesn't provide usage information for the Histogram Metrics License's persisted writes. # Reduce cardinality Source: https://docs.chronosphere.io/control/shaping/shape-metrics/reduce-cardinality Learn how to use the tools provided by Chronosphere Observability Platform to identify and reduce metric cardinality. When you're first using Chronosphere Observability Platform, or a new app or service comes online, you might see cardinality spikes. Cardinality spikes can occur when: * A metric or group of metrics has unexpectedly large numbers of labels. * A process or service creates many similarly named metrics. For more information about cardinality, see [What is high cardinality in observability?](https://chronosphere.io/learn/what-is-high-cardinality/) on the Chronosphere Blog. Cardinality spikes can cause storage and licensing issues. To reduce cardinality, or data storage for less important metrics: 1. Find a problematic metric or label. 2. Review that metric or label's usage. 3. Decide what to do with it (drop, rollup). ## Find a metric and inspect the associated labels Observability Platform provides the following tools to help you understand the impact of metric growth, identify problematic metrics and labels, and assess the impact of existing aggregation rules: * The [Metric Growth dashboard](/observe/dashboards/managed-dashboards#metric-growth) can includes metrics and labels that have recently increased in cardinality. * The [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer) provides real-time insight into current incoming metrics. Sort metrics by **Unique value** to find potential high cardinality. * The [Aggregation Rules UI](/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules) visualizes existing shaping rules and how they affect your environment. Review these rules to understand their impact. If you want to reduce the cardinality of a metric, you must first understand the targeted metric and its associated labels. If you have administrative privileges: 1. In the navigation menu, click ** Go to Admin** and then select ** Analyzers > Live Telemetry**. 2. The analyzer defaults to `_name_`. Sort **Label values** by name, or add a **label filter**. 3. In the **Labels** section, inspect the incoming label keys. The **Unique Values** column shows how many distinct values are incoming for a given label (cardinality), and **Appears In** shows how frequently that label is attached to the metric. When the number of unique values for a metric is high, that label contributes significantly to the cardinality for the metric. ## Review metric and label usage After identifying a high-cardinality label, you need to understand whether this label is meaningful, or if it can be safely removed. To verify dropping a label is safe, use the [](/investigate/analyze/usage#high-volume-low-utility-metrics) to review each label's **Utility score**. This score provides insight into which labels users [find important](/investigate/analyze/usage#usage-patterns). ## Remove the identified label If you identify a label that isn't used in any dashboards or alerts, consider reducing or removing the label using these methods: * Create [drop, mapping, or rollup rules](/control/shaping/shape-metrics/rules#reduce-stored-metrics) to reduce stored metrics by aggregating, downsampling, or dropping unneeded metric data. * Use the [Recommendations](/control/shaping/shape-metrics/reduce-cardinality/recommendations) page to identify metrics and labels with no usage or utility over the past 30 days. Apply the suggested recommendations to reduce the impact on persisted writes and persisted cardinality. * When ingesting Google Cloud metrics, use [filters and aggregations](/ingest/metrics-traces/gcp#control-incoming-metrics). ### Validation For rollup rules, preview the [shaping impact](/control/shaping/shape-metrics/reduce-cardinality/shaping-impact) to review and confirm your changes before deleting metrics and labels that still matter. ## Post validation tasks Return to the Live Telemetry Analyzer and search for your metric. If you've used a rollup rule, it can take some time before your rolled up metric appears. For rolled up metrics, it often makes sense to drop raw data if that data isn't needed. This reduces cardinality and data storage requirements. After you've validated your rule, apply the rule using your selected method. # Review aggregation rules Source: https://docs.chronosphere.io/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules Use the Aggregation Rules page to review your aggregation rules. The Aggregation Rules page is a centralized location in Chronosphere Observability Platform to review all of your [rollup](/control/shaping/shape-metrics/rules) and [mapping](/control/shaping/shape-metrics/rules/mapping) rules. In the navigation menu, click ** Go to Admin** and then select ** Control > Aggregation Rules**. ## Search for a rule Use the **Search rules** field to find a particular rule or group of rules. You can search by: * **Name** * **Filter** * `mapping`, `roll-up`, `keep`, or `discard` Aggregation rules display in a table. Click a column name to sort the table by that column. ## Table display The **Table Settings** button enables you to change the displayed columns. Select a checkbox to add a column, clear the checkbox to remove it. Clear the checkbox for a group to remove all columns in the group. The following columns are available for display: * **Name:** (Default) The rule name and aggregation rule type (rollup or mapping). * **Slug Name:** The unique slug name of the rule. * **Rule Configuration**: This group contains details about the rule. * **Filter:** (Default) Specifies the names of the metrics or labels the rule matches. Filters can include both Prometheus and Graphite metrics. * **Label Policy:** (Default) Indicates whether a rollup rule is discarding specific label names (exclude by) as indicated by the **Discard** label, or explicitly keeping specific label names (group by) as indicated by the **Keep** label. * **Drop Raw:** (Default) Displays `true` when dropping raw values, or `false` when retaining them. * **Metric Name:** (Default) The output metric's name. * **Interval:** Amount of time between aggregated data points. For example, `15s` indicates a 15 second pause between each data storage attempt. * **Metric type:** The [type of metric](/control/shaping/shape-metrics/types). * **Operation:** The [aggregation operation](/control/shaping/shape-metrics/rules/rollup#aggregation-operations) in use. * [**Shaping Impact:**](#shaping-impact) Data relating to rule efficiency. * **Last Modified:** (default) The last time this rule was updated. * **Mode:** The mode the rule is in. Either **Preview** or **Enabled**. For more detail, visit [Fields for rollup rules](/control/shaping/shape-metrics/rules/rollup#rollup-rule-attributes). ### Shaping impact Use *shaping impact* data to manage your traffic shape, by highlighting the benefits of the Control Plane and providing more transparency around the individual impact of existing rules. The following columns explain the **Shaping Impact** of a rule: * **Matched Writes:** The number of writes per second matched or ingested into the Observability Platform aggregation tier. * **Rule Efficiency:** A measure of how effective the rule is in aggregating incoming data points, expressed as a percentage reduction in data points per seconds from incoming to aggregated data. * **Persisted Impact:** The change in total persisted data points per second as a result of the rule. * **Matched Impact:** The percentage of the matched writes license used by the rule. * **Output Metric Utility Score:** An aggregate number that indicates the relative usefulness of the output metric, determined by the number of **References**, **Executions**, and unique users. A higher score means users include the output metric in their workflows. To learn more, refer to [](/investigate/analyze/usage). Some ways you can use shaping impact are: * Reduce persisted data To reduce persisted data in your system, look for rules that have low utility, but add persisted data. Use the **Least Valuable (by Persisted Write)** sort to find rules to change or remove. * Clean up ineffective rules To optimize rules, review rules that aren't aggregating data effectively or don't match incoming data points. Sort by **Least Effective** to find rules to change or remove. ## Sort your data Next to the search box is the sorting menu. These sort options let you view your least or most valuable rules, determined using the underlying shaping impact metrics and your unique needs. By default, rules sort by **Least Effective**. * **Least Valuable (by Matched Impact):** Find rules that have high matched writes, but low output metric utility. * **Most Valuable (by Matched Impact):** Find rules that have low matched writes and high output metric utility. * **Least Valuable (by Persisted Write):** Find rules that add to the persisted data in the system and have low output metric utility. * **Most Valuable (by Persisted Write):** Find rules that conserve persisted data in the system and have high output metric utility. * **Least Effective:** Find rules that either aren't aggregating data effectively or don't match any incoming data points. * **Most Effective:** Find rules that are aggregating data effectively. * **Manual:** Displays when the user clicks a specific shaping impact column to sort by. ## Add aggregation rules Click **Create Aggregation Rule** to display the [rollup rules creation dialog](/control/shaping/shape-metrics/rules/rollup#create-a-rollup-rule). Use the procedure to create a new rollup rule. ## Inspector panel The inspector panel contains tools to help you analyze your metrics and rules. Click any rule to open the panel. ### Shaping impact graphs The **Shaping impact** tab displays historical graphs that show how your rules performed during the selected time frame: * **Matched Writes:** The raw number of incoming data points per second (DPPS) matching the rule. * **Rule Efficiency:** The effectiveness of the rule in aggregating incoming data points. This graph displays the percentage reduction in retained data for the selected rule, and a raw number of **DPPS conserved**. * **System Persisted Impact:** The change in total persisted data points per second as a result of the rule. This graph displays the percentage reduction in retained overall data for the selected rule, and a raw number of **DPPS added** or **DPPS conserved**. * **Matched Impact:** The percentage of the matched writes license used by the rule. Hold the pointer over a graph to display the specific data at point in the graph. Click and drag in the graph to narrow your search to a specific time range. Use the time selector menu to change the period displayed on the graph. Time ranges from the **Last 5 minutes** up to the **Last 14 days**, or you can select a **Custom time range**. To select a custom time range: 1. Select a specific date and time from the picker, or manually enter a **Start Time** and **End Time** in the format `YYYY-MM-DD HH:MM:SS`. 2. Click **Apply** to update the graph, or **Cancel** to close the picker. Click **Refresh** to reload the graphs with the most recent data. ### Rule configuration Click the **Configuration** tab to display the definition used to create the rule. Select **API**, **Chronoctl**, or **Terraform** to set the type of definition displayed. You can copy the definition to your clipboard for use with creating or updating a rule using normal creation methods. ### Analyze incoming metrics or labels You can analyze the incoming data for a specific metric. Select a rule from the list, and then click **View Incoming Metrics**. The [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer) displays, [actively profiling](/investigate/analyze/telemetry-analyzer) the selected metric or value. ### Edit aggregation rules Click **Edit** to display the [rollup rules creation dialog](/control/shaping/shape-metrics/rules/rollup#create-a-rollup-rule). The dialog is pre-populated with the selected rule's details. You can make changes, and then copy the new configuration file and apply it to your system to update the rule using Terraform or Chronoctl. # Apply rule recommendations Source: https://docs.chronosphere.io/control/shaping/shape-metrics/reduce-cardinality/recommendations Use the Recommendations page to identify and reduce metrics and labels with no usage or utility over the past 30 days. Chronosphere Observability Platform includes a **Recommendations** page as part of the Control Plane to help identify metrics and labels with no usage or utility over the past 30 days. The utility score is calculated based on the number of configuration references to the metric, such as dashboards or monitors, and direct query executions, such as in Metrics Explorer. Use this page to understand the impact on persisted writes and persisted cardinality when applying the recommended data optimizations. After applying optimizations, it might take up to 150 minutes to see impacts to cardinality. Impacts to persisted writes are immediate. The aggregation rules on this page incorporate labels that appear together in the total cardinality calculation. ## Drop rule recommendations To apply drop rule recommendations: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Recommendations**. 2. In the left navigation under **Data optimizations**, click the drop rule recommendations. 3. Select one or more metrics to calculate the estimated impact on reducing persisted writes and persisted cardinality. All metrics are selected by default. 4. Click **Create Drop Rule** to create a drop rule definition that contains the selected metrics. 5. In the **Add Drop Rule** drawer, choose whether the rule drops data with any value, or if it meets a particular value. 6. Click **Review** to review your changes. 7. Select the checkbox to indicate that you've reviewed the rule, and then click **Save Rule**. The drop rule is created and displays in the [Drop Rules](/control/shaping/shape-metrics/rules/drop-rules) page. ## Rollup rule recommendations To apply rollup rule recommendations: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Recommendations**. 2. In the left navigation under **Data optimizations**, click the rollup rule recommendations. 3. Select a single metrics to calculate the estimated impact on reducing persisted writes and persisted cardinality. 4. Click **Create Rollup Rule** to create a rollup rule definition that contains the selected metric. 5. Complete the steps to [create a rollup rule](/control/shaping/shape-metrics/rules/rollup#create-a-rollup-rule). # Preview aggregation rule impact Source: https://docs.chronosphere.io/control/shaping/shape-metrics/reduce-cardinality/shaping-impact Use the Shaping Impact Preview to preview the impact of aggregation rules on your system before applying them. Shaping Impact Preview lets you create new aggregation rules to preview the impact of a rule on your overall system. This helps prevent breaking changes and ensure the rules you create operate as expected. A rule created for preview purposes lets you profile the impact of the rule, but isn't enabled in your system and doesn't affect your stored data. Knowing the impact of a rule, and whether it accomplishes the needed task can be difficult. Shaping impact preview uses the existing [rollup](/control/shaping/shape-metrics/rules) or [mapping](/control/shaping/shape-metrics/rules/mapping) aggregation rules to show you how your rule affects your Chronosphere Observability Platform installation before applying the rule and changing your data or stored time series. ## Preview a rule's shaping impact Rules in preview mode display in the [Aggregation Rules UI](/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules) similar to rules already in use. Rules in preview mode use a different background color, and have a **Preview** label next to the rule type in the **Name** column. When a preview rule is added, but hasn't finished impact calculations, the [shaping impact](/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules#shaping-impact) columns display `calculating`. Completed impact calculations display under **Rule Efficiency**, **Matched Impact** (if you're using a matched writes license), and **System Persisted Impact**. **Utility Score** isn't available for rules in preview mode because the **Output metric** can't be used until the rule is enabled. ## Create a preview rule Creating a preview rollup or mapping rule requires adding a configuration variable to the rule creation file called `mode`. The `mode` variable has two states: * `preview`: This rule is in a preview state, and doesn't affect data being mapped, rolled up, or added to storage. * `enabled`: This rule is active and affecting incoming data. If `mode` isn't present, the rule defaults to the `enabled` state. For mapping or rollup rules created with [Chronoctl](/tooling/chronoctl), define `mode` in the `spec` section of the rule definition file. ```text theme={null} mode: PREVIEW ``` For rollup rules created with [Terraform](/tooling/infrastructure/terraform), add: ```text theme={null} # The operational mode for this rule. mode = preview ``` ## Change a rule from preview to enabled Preview rules use the same definition files as active rules. To change a rule from preview to enabled: 1. In the definition file, change the value of `mode` from `preview` to `enabled`. 2. Save the file. 3. Apply the updated definition. The rule begins affecting data, and the **Preview** notice doesn't display in the Aggregation Rules UI. # Metric shaping rules Source: https://docs.chronosphere.io/control/shaping/shape-metrics/rules Learn about the ways to use rules in Chronosphere Observability Platform to shape metric data. Observability Platform applies shaping rules at different phases along the data journey. Drop rules are applied at ingestion time. After metrics are sent but before they're stored, rollup rules and mapping rules are applied. Derived metrics and recording rules are then applied at query time. ```mermaid actions={false} theme={null} flowchart TB subgraph OP["Observability Platform"] subgraph Ingest["Ingestion"] Drop["Drop
rules"] end subgraph Aggregate["Aggregation"] Rollup["Rollup
rules"] Mapping["Mapping
rules"] end subgraph Query["Querying"] Derived["Derived
metrics"] Record["Recording
rules"] end end Ingest --> Aggregate --> Query click Drop "/control/shaping/shape-metrics/rules/drop-rules" click Rollup "/control/shaping/shape-metrics/rules/rollup" click Mapping "/control/shaping/shape-metrics/rules/mapping" click Derived "/investigate/querying/metrics/derived-telemetry/derived-metrics" click Record "/control/shaping/shape-metrics/rules/recording" classDef rules fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF,font-weight:underline; class Drop,Rollup,Mapping,Derived,Record rules ``` ## Reduce stored metrics Chronosphere Observability Platform [ingests](/ingest) metrics from many sources. These sources can supply multiple [metric types](/control/shaping/shape-metrics/types), which are generally ingested as raw data. Although raw data might provide detailed information about a specific point in time, that data grows rapidly and is expensive to store. Use the following shaping rules to control costs by aggregating, downsampling, or dropping unneeded metric data: * [Drop rules](/control/shaping/shape-metrics/rules/drop-rules) reduce incoming data or cardinality to pare down your stored metrics across both time and labels. Use drop rules to omit incoming metrics based on labels, and only retain the metrics you need. These rules help reduce persisted data, and you can implement them quickly. * [Rollup rules](/control/shaping/shape-metrics/rules/rollup) downsample and aggregate metrics after they're sent by the client but before they're stored. Use rollup rules to reduce cardinality, downsample data, and perform basic aggregations across both time and labels. These rules are scalable, can drop raw data, and handle late-arriving data. * [Mapping rules](/control/shaping/shape-metrics/rules/mapping) downsample in-memory metric data on the streaming ingest path and then store any results in the database, which happens before metric aggregation. Use mapping rules to reduce stored metrics across time only. * [Recommendations](/control/shaping/shape-metrics/reduce-cardinality/recommendations) identify metrics and labels with no usage or utility over the past 30 days. Apply the suggested recommendations to reduce the impact on persisted writes and persisted cardinality. When ingesting Google Cloud metrics, use [filters and aggregations](/ingest/metrics-traces/gcp#control-incoming-metrics) to reduce incoming metrics. Review your metrics to determine if they're actionable because they have [*references*](/investigate/analyze/usage#summary), or because some of their labels have high utility (*dimensions*). See [Reducing cardinality](/control/shaping/shape-metrics/reduce-cardinality) for more information about this process. ```mermaid actions={false} theme={null} graph LR accTitle: What kind of rule should I use? accDescr: A flowchart to help with the process of finding metrics to remove. Profile[Profile
metrics
to triage]-->Ref@{shape: diamond, label: "Are there
references?"} Ref -->|Yes| Unused@{shape: diamond, label: "Are there
unused
dimensions?"} Ref -->|No| Drop[Drop the metric] Unused -->|Yes| Rollup[Use rollup
rules to drop
dimensions] Unused -->|No| Value@{shape: diamond, label: "Do the
dimensions
provide value?"} Value --> |Yes| Next[Find next
metric] Value -->|No| Rollup click Profile "/investigate/analyze/telemetry-analyzer" click Drop "/control/shaping/shape-metrics/rules/drop-rules" click Rollup "/control/shaping/shape-metrics/rules/rollup" classDef rules fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF,font-weight:bold; class Profile,Drop,Rollup rules ``` ## Optimize query performance Although Observability Platform provides recording rules to optimize query performance, Chronosphere recommends using [rollup rules](/control/shaping/shape-metrics/rules/rollup) in most cases because they can reduce both cardinality and the volume of the data you persist. [Recording rules](/control/shaping/shape-metrics/rules/recording) primarily improve query performance. You can execute any PromQL query and add labels, which make queries more efficient. However, recording rules impact your persisted data. If you want to change how data is presented to make it easier for users to query, consider using [alias expressions](#alias-expressions), which don't impact persisted data. If you experience issues with [late-arriving data](/control/storage), consider using [rollup rules](/control/shaping/shape-metrics/rules/rollup) instead, since recording rules might impact short-term query results. You can also use rollup rules and drop rules to [fix slow queries](/investigate/querying/metrics/fixing) by reducing the number of time series those queries return. ## Alias expressions [Derived metrics](/investigate/querying/metrics/derived-telemetry/derived-metrics) and [derived labels](/investigate/querying/metrics/derived-telemetry/derived-labels) provide frequently used aliases for queries without impacting persisted data limits. Derived metrics help reduce alert and dashboard complexity, replace [recording rules](/control/shaping/shape-metrics/rules/recording), and provide frequently used aliases for queries. Derived labels are non-destructive, and are specifically designed for efficient operations on time series at scale. Unlike Prometheus relabel rules, which overwrite existing data, derived labels standardize your label names without overwriting them permanently. # Drop rules Source: https://docs.chronosphere.io/control/shaping/shape-metrics/rules/drop-rules Use drop rules to omit metrics you don't need and persist only the metrics that matter. Based on your amount of available storage and rate limits, you might want to reduce the amount of metrics that you continue to store so that you're keeping only those metrics you need. For example, using [](/investigate/analyze/usage), you might find a metric has a low utilization score. Metrics with [high volume and low utilization](/investigate/analyze/usage#high-volume-low-utility-metrics) are candidates for dropping. Use *drop rules* to omit incoming metrics based on labels, and have Chronosphere Observability Platform persist only those metrics you need. Define filters to specify the metric or label names you want to drop. You can also drop metrics based on a data point value. The Collector scrapes data from an app, and sends the data to Chronosphere where you can apply drop rules. Any metrics that weren't impacted by drop rules are then aggregated in Observability Platform. ```mermaid actions={false} theme={null} graph LR accTitle: Drop rules data flowchart accDescr: Where drop rules take effect in the flow of data into Observability Platform. col(Collector) -- Scrapes --> app(Application) col -- Sends --> Drop("Drop rules") subgraph OP["Observability Platform"] Drop -- Metrics not dropped --> ca(Aggregate metrics) end ``` Drop rules affect data from the point of rule creation onward. When you query for the metric: * Data can display in autocomplete searches for a period of approximately two hours. * Queries covering the time period before the drop rule took effect display the metric until the [retention period](/administer/limits-licensing/licensing#contract-details) expires. ## View and filter created drop rules To view the drop rules defined on your instance, you must have administrative privileges. Select from the following methods to view your available drop rules. To view drop rules: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Drop Rules**. 2. Use the **Search rules** text box to find a specific rule. This search looks for any text in any rule. 3. Select a value from the **Mode** menu to filter for rules matching the selected mode. 4. Select a drop rule from the list that you want to view. 5. Optional: To annotate a drop rule with additional context, on the **Edit Drop Rule** panel click the **Comments** tab to [add a comment](/navigate/comments). For information about viewing, copying, or downloading rule configurations, see [Rule configuration](/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules#rule-configuration). To complete this action with the Chronosphere API, use the [`ListDropRules`](/tooling/api-info/definition/operations/ListDropRules) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create a drop rule Use drop rules to drop the ingestion of certain metrics that you don't need to persist. For example, dropping metrics from a specific cluster, or test data. Drop rules can have multiple filter conditions on different labels, making it possible to drop a subset of the series matching a particular metric name. Use multiple filters for cases where only a subset of a metric's data is used. For example, if a metric is used in an alert but examines only series related to error statuses, you can drop non-error status series. You can use [glob syntax](#matching-with-glob-syntax) in your drop rule to match specific values. Use the `value_based_drop` object to define a [value-based drop rule](#define-a-value-based-drop-rule) that drops metrics based on a target data point value. Drop rules have these definition limitations: * The rule can't be longer than 8,192 characters. * A drop rule `slug` must contain only alphanumeric characters, dashes, or underscores. The allowed regular expression for matching is `[a-zA-Z0-9-_]`. Spaces aren't allowed. * Each rule `name` must be unique. After creating a drop rule, it takes roughly one minute for the rule to take effect. Wait at least one minute after creating a drop rule before making updates to it. A metric that matches any drop rule is dropped, regardless of overlap with drop rules created using other methods. For example, if a metric matches a drop rule created with Terraform that also matches a subset of a drop rule created with the [`CreateDropRule`](/tooling/api-info/definition/operations/CreateDropRule) endpoint, the metric is dropped. A drop rule containing `kubernetes_cluster:dev` matches and drops lower value metrics containing this key-value pair. To exclude specific values that also match this rule, use a single `filter` with an `AND` logical operator to specify a "does not equal" match. The following drop rule matches any metric containing `kubernetes_cluster:dev`, but not metrics that also contain the specified `container_name` values: ```text theme={null} "kubernetes_cluster:dev AND container_name:!{log_service|data_service}" ``` Select from the following methods to create a drop rule. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To create a drop rule, you must have administrative privileges: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Drop Rules**. 2. Click **Create drop rule**. 3. In the **Create drop rule** drawer, select a **Mode** for the drop rule, which can be one of the following values: * **Enabled**: Create the rule and make it active immediately. * **Preview**: Create the rule and evaluate it, but don't drop data. * **Disabled**: Create the rule, but don't evaluate it to drop data. 4. Enter a **Name** for the rule. 5. Enter one or more **Matchers**, which are labels with or without a specific value. You can use [glob syntax](/investigate/querying/glob-syntax) in the **Value** field. To specify multiple values, separate values with a comma and surround them with curly braces (`{}`). For example, `key={value1,value2,value3}`. To match values containing a comma, specify the values without braces. For example, `key=value,continue`. When allowing only particular metrics, you might need a double negation, which requires specific logical filtering: ```text theme={null} keep: a==A && b==B drop: !(a==A && b==B) ``` In combination, this logic leads to a drop rule where `!a==A || !b==B --> a!=A || b!=B`. Comma-separated lists can't include spaces after commas. Rules with a space after a comma don't evaluate. 6. To define a [value-based drop rule](/control/shaping/shape-metrics/rules/drop-rules#define-a-value-based-drop-rule), choose **0** or **1** in the **Select drop value** menu. 7. Click **Review**. 8. Review the rule to ensure it's correct. 9. Select the checkbox for **I have reviewed the proposed changes and they look correct**. 10. Click **Save rule** to save the rule, or click **Back** to change the rule. To create a drop rule with [Chronoctl](/tooling/chronoctl): 1. Run the following command to generate a sample drop rule configuration you can use as a template: ```shell theme={null} chronoctl drop-rules scaffold ``` You can redirect this output to a file for editing: ```shell /FILE_NAME/ theme={null} chronoctl drop-rules scaffold > FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML definition file you want to use. 2. Configure the rule's properties in the YAML resource definition. See the [Chronoctl example](#chronoctl-example) for a completed definition. 3. With a completed definition, submit it with: ```shell /FILE_NAME/ theme={null} chronoctl apply -f FILE_NAME.yaml ``` When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. To create a drop rule with [Terraform](/tooling/infrastructure/terraform): 1. Create or edit a Terraform file and add the definition by using the `chronosphere_drop_rule` type, followed by a name in a resource declaration. 2. Run this command to apply the changes: ```shell theme={null} terraform apply ``` See the [Terraform example](#terraform-example) for a completed drop rule resource. To complete this action with the Chronosphere API, use the [`CreateDropRule`](/tooling/api-info/definition/operations/CreateDropRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Chronoctl example The following resource defines a drop rule named `Drop node exporter` in `PREVIEW` mode, with the slug `drop-node-exporter-metrics`: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: DropRule spec: name: "Drop node exporter" # Unique identifier for rule slug: "drop-node-exporter-metrics" filters: - name: job value_glob: node_exporter # Boolean to determine if rule is currently active mode: PREVIEW value_based_drop: enabled: true # Configuration of which metric values to drop; must be 0 or 1 target_drop_value: 0 # Boolean to determine whether to drop data points that are NaN. drop_nan_value: true ``` ### Terraform example The following definition creates a drop rule that Terraform refers to as `drop-node-exporter`, with the name `Drop node exporter`. The rule drops all metrics from the `node_exporter` job. ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_drop_rule" "drop-node-exporter" { # Name for the rule name = "Drop node exporter" # Unique identifier for rule; generated automatically if left blank slug = "drop-node-exporter-metrics" # Label matcher to match metrics that should be dropped; all labels must match query = [ "job=node_exporter", ] # Whether the rule is enabled; one of ENABLED, DISABLED, or PREVIEW mode = "ENABLED" # Boolean to determine whether to drop data points that are NaN. drop_nan_value = true } ``` This rule is an example of a drop rule based on the value of the incoming data point: ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_drop_rule" "value-based-drop-rule" { # Name for the rule name = "Drop Rule" # Label matcher to match metrics that should be dropped; all labels must match query = [ "ab=123", ] # Whether the rule is enabled; one of ENABLED, DISABLED, or PREVIEW mode = "ENABLED" # Configuration of which metric values to drop; must be 0 or 1 value_based_drop { target_drop_value = 1 } # Boolean to determine whether to drop data points if the values are NaN drop_nan_value = true } ``` ### Matching with glob syntax Drop rules support glob syntax, but don't support PromQL operators in the `value_glob` field to indicate negation. The following glob syntax applies to drop rules: * To match on any value, include an asterisk (`*`) in the `value_glob` field. For example, the following rule matches values that start with `envoy_cluster_grpc` for the `__name__` metric ```yaml theme={null} filters: - name: __name__ value_glob: envoy_cluster_grpc* ``` * To not match on a value for a metric label, include an exclamation mark (`!`) in the `name` field. For example, the following rules match any metrics that don't have a `container` label with a value of `database-backup` or `database-restore`: ```yaml theme={null} filters: - name: container! value_glob: database-{backup,restore} ``` Verify your [glob syntax](/investigate/querying/glob-syntax) with [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer) to ensure your query matches the correct metrics. You can also [create drop rules](/control/shaping/shape-metrics/rules/drop-rules#create-a-drop-rule) in preview mode to evaluate a rule without dropping data. This capability lets you test a rule without impacting your persisted data. #### Glob syntax examples The following definition for a drop rule in Chronoctl includes a few different rules that include matching on any value (`*`) and negation syntax (`!`): * Drops any metric that doesn't have an `id` field with a forward slash (`/`) as a value. * Drops any metric that includes the `pod` key, with any value. * Drops any metric that doesn't have the `container` field, with any value. ```json theme={null} { "drop_rule": { "slug": "cAdvisor-pause-containers", "name": "cAdvisor pause containers", "mode": "ENABLED", "filters": [ { "name": "job", "value_glob": "k8s-cadvisor" }, { "name": "id!", "value_glob": "/" }, { "name": "pod", "value_glob": "*" }, { "name": "container!", "value_glob": "*" } ] } } ``` The following drop rule for the [CreateDropRule API](/tooling/api-info/definition/operations/CreateDropRule) drops any metric containing the `service` label with a value of `api` that also has a `cluster` label that begins with `dev`. ```json theme={null} { "drop_rule": { "slug": "api-drop-rule", "name": "api drop rule", "mode": "ENABLED", "filters": [ { "name": "service", "value_glob": "api" }, { "name": "cluster", "value_glob": "dev*" } ] } } ``` To keep only metrics with a specific label value, use glob syntax to specify a "does not equals" match with an exclamation mark `!` in the `name` field to match specified values. For example, the following drop rule drops any metric containing the `service` label with a value of `api`, that doesn't also have an `env` label with a value of `production`. This drop rule also applies if the `env` label isn't set. ```json theme={null} { "drop_rule": { "slug": "api-drop-rule", "name": "api drop rule", "mode": "ENABLED", "filters": [ { "name": "service", "value_glob": "api" }, { "name": "env!", "value_glob": "production" } ] } } ``` A label with an empty value is the same as a label that isn't present. Use an asterisk `*` in the `value_glob` field to match metrics with a particular label, regardless of the value. Conversely, to drop only metrics that don't have a particular label, use an exclamation mark `!` in the `name` field to specify a "does not equal" match, in conjunction with an asterisk `*` to match all values. For example, the following drop rule drops any metric containing the `service` label with a value of `api` that doesn't also have an `env` label. ```json theme={null} { "drop_rule": { "slug": "api-drop-rule", "name": "api drop rule", "mode": "ENABLED", "filters": [ { "name": "service", "value_glob": "api" }, { "name": "env!", "value_glob": "*" } ] } } ``` Observability Platform glob syntax doesn't support using two asterisks where one of them is in the middle of a string. For example, `*k8s*staging` isn't valid. ### Define a value-based drop rule Use the `value_based_drop` object to define a value-based drop rule. The `target_drop_value` parameter specifies the target data point value at which to drop metrics. The `value_based_drop` parameter drops time series based on data point values of `0` or `1`. This type of drop rule can significantly reduce data points per second, but changes metrics from continuous series to sparse metrics. Adding a rule of this type can change the behavior of existing monitors or dashboards that use the target metric. You can include the `drop_nan_value` parameter to drop data point values that are Not a Number (NaN). If set to `true`, Chronosphere drops NaN data points, along with any published [staleness markers](/ingest/metrics-traces/collector/configure/optimizations#enable-staleness-markers). If you set the `drop_nan_value` parameter to `true` in conjunction with the `value_based_drop` parameter, Observability Platform drops data points matching the value *or* any NaN values. If you want to drop *only* NaN values, set `drop_nan_value` to `true` without a corresponding `value_based_drop` parameter. When no value is set for `drop_nan_value`, all values are dropped, including NaN values. ## View drop rule impact After [creating a drop rule](#create-a-drop-rule), query the [`chrono_metrics_drop_rule_dpps_matched`](/overview/concepts/dictionary#impact-statistics) metric to understand the impact of your drop rule. Use the labels this metric provides to answer these questions: * Which data points were dropped by active drop rules? * Which active rules overlap or duplicate other drop rules? * What are the total matches across all rules? To understand the impact of a drop rule: 1. In the navigation menu select ** Explorers > Metrics Explorer**. 2. In the **Metrics Explorer** page, click **Builder** and then in the **Metric** field, enter the following metric name: ```text theme={null} chrono_metrics_drop_rule_dpps_matched ``` 3. Click **Run** to run the query. The results include all drop rules. 4. Click the **Filters** field. In the **Labels** subfield, select `slug` as the label to match on, and then click the **Value** subfield to search for the name of your drop rule. 5. Click **Run** to run the updated query and narrow the results. 6. In the **Aggregation** field, select **Sum** as the aggregation, and then select **applied** and **mode** as the values to aggregate by. 7. Click **Run** to run the updated query. You can add filters to narrow the results to answer specific questions. The following examples provide some common questions, and include the query to run in Metrics Explorer to answer each question: * **How can I preview how much data a given drop rule will drop?** ```text /SLUG/ theme={null} sum(chrono_metrics_drop_rule_dpps_matched{mode="preview", slug="SLUG"}) ``` Replace `SLUG` with the slug of your drop rule. This metric creates the drop rule in `preview` mode. You can then use the following metric to enable the drop rule and apply it. * **How much data does an existing, enabled rule drop?** ```text wrap /SLUG/ theme={null} sum(chrono_metrics_drop_rule_dpps_matched{mode="enabled", applied="true", slug="SLUG"}) ``` Replace `SLUG` with the slug of your drop rule. * **How much data am I actually dropping across my existing drop rules?** ```text wrap /SLUG/ theme={null} sum by (slug) (chrono_metrics_drop_rule_dpps_matched{mode="enabled", applied="true"}) ``` * **What's the overall combined number of data points dropped as a result of my enabled and preview rules?** ```text /SLUG theme={null} sum(chrono_metrics_drop_rule_dpps_matched{applied="true"}) ``` * **Which rules aren't effective because they're duplicates of existing rules that are dropping data?** ```text wrap /SLUG/ theme={null} sum by (slug) (chrono_metrics_drop_rule_dpps_matched{mode="enabled", applied="false"}) ``` ## Disable a drop rule Select from the following methods to disable a drop rule. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To disable an existing drop rule in Observability Platform, you must have administrative privileges: 1. In the navigation menu, click ** Go to Admin** and then select ** Control > Drop Rules**. 2. Select the rule you want to disable from the list. 3. Toggle the rule to the **Disabled** state. 4. Click **Save**. To re-enable the rule, use the previous steps and instead toggle **Always Active**. To disable a drop rule with [Chronoctl](/tooling/chronoctl): 1. In the drop rule specification, set `mode` to `DISABLED`. ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: DropRule spec: name: "Drop node exporter" # Unique identifier for the rule slug: "drop-node-exporter-metrics" filters: - name: job value_glob: node_exporter # Boolean to determine if rule is currently active mode: DISABLED value_based_drop: enabled: true target_drop_value: 0 ``` 2. Submit the disabled drop rule definition to Observability Platform: ```shell theme={null} chronoctl apply -f FILE.yaml ``` To disable a drop rule with [Terraform](/tooling/infrastructure/terraform): 1. Edit the drop rule's definition in a Terraform file to set `active` to `false`. 2. Run the following command: ```shell theme={null} terraform apply ``` For example, the following definition disables an always active drop rule that Terraform refers to as `drop-node-exporter`, with a human-readable name of `Drop node exporter`. ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_drop_rule" "drop-node-exporter" { # Name for the rule name = "Drop node exporter" # Unique identifier for the rule slug = "drop-node-exporter" # Label matcher to match metrics that should be dropped; all labels must match query = [ "job=node_exporter" ] # Set to DISABLED to stop the rule from dropping metrics mode = "DISABLED" } ``` To complete this action with the Chronosphere API, use the [`UpdateDropRule`](/tooling/api-info/definition/operations/UpdateDropRule) endpoint and update the `mode` key to `DISABLED`. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete a drop rule Select from the following methods to delete a drop rule. 1. In the navigation menu select **Control > Drop Rules**. 2. Select the rule you want to delete from the list. 3. Click the delete icon to delete the rule. To delete a drop rule using [Chronoctl](/tooling/chronoctl), pass the rule's slug to the `drop-rules delete` command: ```shell theme={null} chronoctl drop-rules delete SLUG ``` Replace *`SLUG`* with the slug of the rule you want to delete. To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteDropRule`](/tooling/api-info/definition/operations/DeleteDropRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # Find and fix rule and monitor evaluation failures Source: https://docs.chronosphere.io/control/shaping/shape-metrics/rules/evaluation-failures View and fix recording rule and monitor evaluation failures. [Recording rules](/control/shaping/shape-metrics/rules/recording) and [monitors](/investigate/alerts/monitors) might not evaluate properly, leaving you with missing data or errors that weren't caught. ## View recording rules and monitor failures Select from the following methods to view your recording rule or monitor failures. In the navigation menu, click ** Go to Admin** and then select **Platform > Rule Status**. The **Rule Status** page displays. Select **Monitors** or **Recording Rules** to focus on a set of errors. Each type's display includes the: * **Time Frame**: The amount of time data aggregated. Permanently set to **last 5 minutes**. * **Total Monitors** or **Total Recording Rules**: The number of this type of definition. * **Failing Monitors** or **Failing Recording Rules**: The number of monitors or rules currently failing to execute. * **Go To Recording Rules**: On the **Recording Rules** tab, this link goes to the recording rules page. The page provides a table with the following information: * **Execution Status** of the rule in the **Time Frame**. * **Monitor** or **Recording Rule** name. * **Interval** the rule evaluates at. * **#Errors** shows the number of failed evaluations in the **Time Frame**. * **Error** text explaining the failure. At the end of the line, click the three vertical dots icon and then a menu option to: * **View Full Error Text**: Review the error text in a dialog box. * **Copy Error Text**: Copy the text of the error message. * **Go to Monitor**: For monitor failures, go to the failed monitor. To view rule evaluation failures with [Chronoctl](/tooling/chronoctl), use the `rule-evaluations list` command: ```shell theme={null} chronoctl rule-evaluations list ``` The output contains all the rule evaluation failures that occurred within the last five minutes, including the following information: * `rule_slug` * `rule_type`: What type of rule (or monitor) this error is for. * `detected_at`: Rule evaluation issues aggregated over the last five minutes, leading up to this timestamp. * `count`: How many failures occurred in the time period. * `message`: The error message, as explained in [common failures and solutions](#common-failures-and-solutions). For example: ```shell theme={null} chronoctl rule-evaluations list ``` The output looks similar to the following: ```text theme={null} rule_slug: bad-rule rule_type: MONITOR detected_at: 2023-02-27T22:44:39.884Z count: 20 message: vector contains metrics with the same labelset after applying alert labels --- rule_slug: bad-rule-2 rule_type: RECORDING detected_at: 2023-02-27T22:44:39.884Z count: 5 message: An internal error occurred. ``` See the [List Rule Evaluation](/tooling/api-info/state_definition/operations/ListRuleEvaluations) API documentation to learn more about these individual fields. To complete this action with the Chronosphere API, use the [`ListRuleEvaluations`](/tooling/api-info/state_definition/operations/ListRuleEvaluations) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Delete low value rules Use the [](/investigate/analyze/usage) to review metrics used in failing rules or monitors. If the metric is low value, deleting the failing rule or monitor might make more sense than fixing it. ## Common failures and solutions The following list provides some common errors and solutions to help you fix failing rules and monitors. ### Vector contains metrics with the same label set after applying labels Prometheus requires all time series returned from a monitor query or a reporting rule be fully unique, meaning the entire set of `label:value` pairs must differ across a time series. If metrics have the same labels after applying alert or rule labels, a collision occurs. Similar to Prometheus, Observability Platform takes monitor or recording rule labels and overrides the label pairs from all returned time series. The following error message indicates a monitor or recording rule `label:value` collision: ```text theme={null} Vector contains metrics with the same label set after applying alert (or rule) labels` ``` For example: * Monitor or Recording Rule labels: `{"service": "gateway"}` * Fetched time series labels: `{"action": "http_get", "service": "ui-console"}` * Resulting time series: `{"action": "http_get", "service": "gateway"}` In this instance, the `ui-console` label is overridden to `gateway` after the monitor or recording rule labels apply. The error occurs in a situation where collected values look like: * Monitor or Recording Rule labels: `{"service": "gateway"}` * Fetched time series 1: `{"action": "http_get", "service": "ui-console"}` * Fetched time series 2: `{"action": "http_get", "service": "backend-server"}` Processing rewrites these time series to: * Resulting time series 1: `{"action": "http_get", "service": "gateway"}` * Resulting time series 2: `{"action": "http_get", "service": "gateway"}` After applying the monitor or recording rule override `{"service": "gateway"}`, the resulting time series are an exact match, which causes an error. Use one of the following methods to resolve the error: * Use the Prometheus [`label_replace`](https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace) operator to change the underlying label name being overwritten. * Remove the monitor or recording rule label. ### Found duplicate time series The error message `Found duplicate series for the match group` indicates two time series being joined together, but the series don't have the same labels. For example, one time series might have a [host or instance label](/ingest/metrics-traces/collector/mappings/prometheus/prometheus-recommendations#label-names-and-values), while the other doesn't. Review the error message and identify the problematic labels. Use one of the following methods to address this issue: * Remove the labels from the offending metric. If the labels aren't used in dashboards, monitors, recording rules, or queries, you can create a rollup rule to remove the labels from the metric. * Update the query to exclude the problematic labels. If other resources use the label, or you want to keep the labels for any other reason, update the query using PromQL functions, such as `group`, `sum`, or `max`, and use the `without` option to exclude the labels. For example, `group(test1) without (host, instance)`. Refer to the [PromQL documentation](https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators) for the behavior of each function. ### Template errors Invalid Prometheus query templates display errors like `undefined variable `"\$labels"\`. Observability Platform attempts to parse your queries using [go template syntax](https://pkg.go.dev/text/template). This error typically means you have a block that looks like `{{ }}` somewhere in the raw query. Remove those blocks to fix this issue. ### The query exceeded the allowable resource limit Resource exhaustion occurs when a query has requested more time series than system resources can support. For example, a query that returns millions of results exceeds the [query scale protections](/administer/limits-licensing/limits/query-limits#query-scale-protections) defined by limits the system can process. Use one of the following methods to address this issue: * Reduce the number of time series returned by the query by adding more label filters. This might not return all of the results you need, so you might need to write multiple recording rules and then update your dashboards and monitors to use the appropriate metric. * Observability Platform provides [rollup rules](/control/shaping/shape-metrics/rules/rollup) you can use to remove labels from metrics and aggregate values together, reducing cardinality. Rollup rules can dramatically reduce the number of time series for a particular metric, which might let your query complete. ### Context deadline exceeded This error is functionally similar to [queries exceeding the allowable resource limit](#the-query-exceeded-the-allowable-resource-limit). Correct these errors with the same solutions. ### Internal error occurred It's possible that when [viewing recording rules](#view-recording-rules-and-monitor-failures) in Observability Platform, the following error could display: ```text theme={null} An internal error occurred. ``` This error can indicate that a metric that's impacted by a recording rule can't be written to the database. A likely cause is that a tenant is experiencing license penalization. Review the [License Overview](/administer/limits-licensing/licensing) to track your telemetry usage against license quotas and determine which license is exceeding its capacity. # Mapping rules Source: https://docs.chronosphere.io/control/shaping/shape-metrics/rules/mapping Learn about how and when to use mapping rules before storing any results in the database. Mapping [aggregation rules](/control/shaping/shape-metrics/rules) downsample in-memory metric data on the streaming ingest path before storing any results in the database, which is before metric aggregation. Downsampling requires the creation of two mapping rules: * A rule which performs the downsampling * A rule that deletes the raw metric If you don't create a mapping rule to drop the raw metric, two metrics persist; one with the original resolution and one with the new resolution. Mapping rules drop a given set of metrics by using the `drop: true` key-value pair. This tactic is often paired with rollup rules, because often the goal of a rollup rule is to remove the original raw metrics after aggregation. This is unlike [drop rules](/control/shaping/shape-metrics/rules/drop-rules), which drop the metric before any aggregation takes place. This achieves the same result as setting the `drop_raw: true` flag on the rollup rule; if this flag is set, it's not necessary to add a matching mapping rule with `drop: true`. Mapping rules support both Prometheus and Graphite metrics. ## When to use mapping rules Chronosphere recommends using mapping rules instead of rollup rules in scenarios where you're ingesting data points for a time series at a sporadic rate, and that exceeds the standard `15s`, `30s`, and `60s` [resolutions](#define-a-metrics-resolution). These denser time series consume more query resources, which means they use more data points per second (DPPS) compared to the amount of cardinality you're storing. Both mapping rules and rollup rules get applied in the same aggregation phase, which means that mapping rules consume matched writes. ## View mapping rules Select from the following methods to view your mapping rules. To list your existing mapping rules using [Chronoctl](/tooling/chronoctl), use the `chronoctl mapping-rules list` command: ```shell theme={null} chronoctl mapping-rules list ``` To complete this action with the Chronosphere API, use the [`ListMappingRules`](/tooling/api-info/definition/operations/ListMappingRules) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create a mapping rule Select from the following methods to create a mapping rule. To create a mapping rule with [Chronoctl](/tooling/chronoctl), define the rule in a YAML file and apply it. If you don't already have a YAML configuration file, use the `scaffold` Chronoctl parameter to generate a template for a specific resource type: ```shell theme={null} chronoctl mapping-rules scaffold ``` You can redirect the results (using the redirection operator `>`) to a file for editing. 1. Create or edit a YAML configuration file to configure the mapping rule. 2. Apply the mapping rule: ```shell theme={null} chronoctl apply -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML configuration file. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. Create a mapping rule with Terraform by using the `chronosphere_mapping_rule` type, followed by a name in a resource declaration. The following mapping rule defines a metric retention interval for a Kubernetes pod: ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_mapping_rule" "http_request_duration" { name = "http request duration" filter = "__name__:http_request_duration k8s_pod:*" aggregations = [ "LAST", ] storage_policy { resolution = "30s" retention = "120h" } mode = "ENABLED" } ``` 1. Add the definition to a Terraform file. 2. Run this command to create the resource: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`CreateMappingRule`](/tooling/api-info/definition/operations/CreateMappingRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. Mapping rules take effect immediately but might require a full recording interval to show a change. Here are some example mapping rules. ### Define a metric's resolution Although you can adjust the resolution for a mapping rule and configure the interval between aggregated data points, Chronosphere doesn't recommend this approach for the sole purpose of reducing resolution. Instead, adjust the scrape or push interval in the [Chronosphere Collector](/ingest/metrics-traces/collector/discover/scrape-configuration) or the [OpenTelemetry Collector](/ingest/metrics-traces/otel/otel-ingest#send-operational-metrics-about-the-opentelemetry-collector), depending on how you send metrics to Observability Platform. The following example uses these keys: * `aggregation_policy`: Specifies the policy applied to aggregate the metric for a given resolution window. * `aggregation`: For example, setting this property to `LAST` takes the last element as the aggregated metric. See [Supported aggregation operations](/control/shaping/shape-metrics/rules/rollup#aggregation-operations) for a complete list. * `interval`: Specifies the time between aggregated data points. Intervals are based on your [retention policy](/administer/limits-licensing/licensing#contract-details). * `filters`: Specifies that the rule catches metrics of name `http_request_duration` with the label `k8s_pod`. Label filters can include multiple labels. Metrics must match each label for the filter to apply. Label values support [glob syntax](/investigate/querying/glob-syntax), including matching multiple patterns with an OR, such as `pattern1,pattern2`. ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: MappingRule spec: name: http request duration slug: http_request_duration aggregation_policy: aggregation: LAST interval: 30s filters: - name: "__name__" value_glob: "http_request_duration" - name: "k8s_pod" value_glob: "*" ``` ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_mapping_rule" "http_request_duration" { name = "http request duration" filter = "__name__:http_request_duration k8s_pod:*" aggregations = [ "LAST", ] storage_policy { resolution = "30s" retention = "120h" } mode = "ENABLED" } ``` ### Drop a metric You don't need to specify an aggregation operation or interval to drop a metric. ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: MappingRule spec: filters: - name: "__name__" value_glob: http_request_bucket - name: k8s_pod value_glob: "*" - name: le value_glob: "*" - name: git_sha value_glob: "*" - name: route value_glob: "*" name: drop raw http_request_bucket slug: http_request_bucket drop: true ``` ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_mapping_rule" "http_request_bucket" { name = "drop raw http_request_bucket" filter = "__name__:http_request_bucket k8s_pod:* le:* git_sha:* route:*" aggregations = [ "LAST", ] drop = true mode = "ENABLED" } ``` ## Delete a mapping rule Select from the following methods to delete a mapping rule. To delete a mapping rule using [Chronoctl](/tooling/chronoctl), use the `chronoctl mapping-rules delete` command: ```shell theme={null} chronoctl mapping-rules delete SLUG ``` Replace *`SLUG`* with the slug of the rule you want to delete. To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteMappingRule`](/tooling/api-info/definition/operations/DeleteMappingRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # Recording rules Source: https://docs.chronosphere.io/control/shaping/shape-metrics/rules/recording Learn about recording rules and when to use them, along with their limitations. Recording rules are a type of [aggregation rule](/control/shaping/shape-metrics/rules) used primarily to improve the performance of frequent queries. Recording rules let you compute frequently used or expensive queries ahead of time, and save the results as a new set of time series. Instead of running the expensive or complex query, Chronosphere Observability Platform queries for the time series generated by the recording rule. This concept is similar to using a precomputed lookup table to avoid complex calculations and find data faster. Recording rules let you alias repeated expressions used in dashboards, and also alias other recording rules. For example, a recording rule might compute the total usage percentage for a resource, which can be referenced in other recording rules and dashboards. Recording rules run based on a fixed interval. They ingest raw metric data into the database before reading it, save the results in a new time series, generate the aggregated and downsampled metric data, and store the data in the database. ## Limitations Although recording rules are powerful, they have the following limitations: * Recording rules might be delayed because they run in a batch format. * Recording rules apply only to individual metrics, not broad aggregations. * Recording rules don't have the ability to discard raw data after aggregation. * Recording rules don't capture [late-arriving data](/administer/limits-licensing/limits/metric-limits#late-arriving-metrics). * If a recording rule fails to run, there's no way to backfill the data. To avoid these limitations, use [rollup rules](/control/shaping/shape-metrics/rules/rollup) or [derived telemetry](/investigate/querying/metrics/derived-telemetry) instead. For example, if you have issues with late-arriving data, consider using rollup rules instead of recording rules. ## Attributes Recording rules support only Prometheus metrics. Configure a recording rule with a PromQL statement executed against the metrics data with the result stored in a new time series with a unique metric name. PromQL statements in recording rules can include any PromQL function. See the [Recording rule API](/tooling/api-info/definition/operations/CreateRecordingRule) for a full definition and list of attributes. ## Best practices Recording rules support adding labels to the resultant aggregated metrics, which rollup rules don't support. Rollup rules also require using either a Prometheus relabel rule, or a derived metric with a `label_replace` function in conjunction with the rollup rule, to accomplish the same goal. Due to architectural differences between Observability Platform and Prometheus, defining recording rules is sometimes different, especially for expensive recording rules that span many metrics. For example, recording rules in Observability Platform are part of a rule group, whereas recording rules in Prometheus aren't guaranteed to be run sequentially. Observability Platform uses a single data store. To enhance performance, use the following recommendations: * Break up the recording rules to scope to different clusters, or another label that scopes your metrics. * Use the `metric_name` field so they all get written back into the same name. With a Prometheus or Thanos setup, Chronosphere recommends [scoping the rules to the local Prometheus server](https://thanos.io/tip/components/rule.md/#rule-aka-ruler) to avoid cross-Prometheus queries. ## View recording rules Select from the following methods to view your recording rules. In the navigation menu, click ** Go to Admin** and then select ** Control > Recording Rules**. The recording rules page is searchable by rule **Name** or **Execution group**. The following fields display: * **Name:** The rule name. * **Execution Group:** The execution group this rule is assigned to. Rules in the same execution group run at intervals. The entire group must complete an execution before the rules in that group will run again. * **Metric Name:** The time series to output to. * **Interval:** How often the rule evaluates. * **Labels:** Label names added to the output metric. * **Query:** Click the `<>` to display the query used for this rule. To list your existing recording rules using [Chronoctl](/tooling/chronoctl), use the `chronoctl recording-rules list` command: ```shell theme={null} chronoctl recording-rules list ``` To complete this action with the Chronosphere API, use the [`ListRecordingRules`](/tooling/api-info/definition/operations/ListRecordingRules) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Create or update a recording rule Select from the following methods to create or update recording rules. Users can modify Terraform-managed resources only by using Terraform. [Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources). To create a recording rule with [Chronoctl](/tooling/chronoctl), define the rule in a YAML file and apply it. If you don't already have a YAML configuration file, use the `scaffold` Chronoctl parameter to generate a template for a specific resource type: ```shell theme={null} chronoctl recording-rules scaffold ``` You can redirect the results (using the redirection operator `>`) to a file for editing. 1. Create or edit a YAML configuration file to configure the recording rule. See the [Chronoctl example](#chronoctl-example) for a complete configuration file. 2. Apply the recording rule: ```shell theme={null} chronoctl apply -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML configuration file. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. Create a recording rule with [Terraform](/tooling/infrastructure/terraform) by using the `chronosphere_recording_rule` type followed by a name in a resource declaration. For more information, see the [Terraform recording rule example](#terraform-example). 1. Add the definition to a Terraform file. 2. Run this command to create the resource: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`CreateRecordingRule`](/tooling/api-info/definition/operations/CreateRecordingRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Chronoctl example The following YAML example includes three recording rules that calculate the average rate of increase per second for jobs that contain a value for `node`. The results display for `instance` and `container` as measured over one minute. This example uses the `metric_name` field to specify the output name of the time series, and the `name` field to display the human readable name. For backwards compatibility, the example uses the `name` field for the time series if `metric_name` isn't specified, like in the third rule. ```yaml expandable Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: RecordingRule spec: name: cpu-usage-seconds-sum-rate-1m slug: instance-container-cpu-usage-seconds-sum-rate1m prometheus_expr: sum(rate(container_cpu_usage_seconds_total{node=""}[1m])) by (instance, container) metric_name: instance_container:cpu_usage_seconds:sum_rate1m interval_secs: 60 label_policy: add: resource: cpu --- api_version: v1/config kind: RecordingRule spec: name: network-receive-bytes-sum-rate-1m slug: instance-container-network-receive-bytes-sum-rate1m metric_name: instance_container:network_receive_bytes:sum_rate1m prometheus_expr: sum(rate(container_network_receive_bytes_total{node=""}[1m])) by (instance, container) interval_secs: 60 label_policy: add: resource: network-receive --- api_version: v1/config kind: RecordingRule spec: name: instance_container:network_transmit_bytes:sum_rate1m slug: instance-container-network-transmit-bytes-sum-rate1m prometheus_expr: sum(rate(container_network_transmit_bytes_total{node=""}[1m])) by (instance, container) interval_secs: 60 label_policy: add: resource: network-transmit ``` ### Terraform example The following code creates a recording rule that Terraform refers to as `scrape_duration_recording_rule` with the name `60s rule`, and defines the other data needed to create a recording rule. ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_recording_rule" "scrape_duration_recording_rule" { # Name for the rule # Value forms the metric name output if metric_name is not specified. name = "60s rule" metric_name = "scrape_duration_seconds:max_60s" # Arbitrary labels to attach to the rule # These labels end up being part of the output metric, # and take precedence over any labels that the expression would have created. # For example, if the expression had metrics with "foo=bar", but the value was # "foo=test", the final metric generated would have "foo=test". labels = { "owner" = "infra" } # Interval at which to evaluate the rule interval = "60s" # The PromQL expression to evaluate expr = "max(scrape_duration_seconds)" } ``` ## Delete a recording rule Select from the following methods to delete a recording rule. To delete a recording rule using [Chronoctl](/tooling/chronoctl), use the `chronoctl recording-rules delete` command: ```shell theme={null} chronoctl recording-rules delete SLUG ``` Replace *`SLUG`* with the slug of the rule you want to delete. To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteRecordingRule`](/tooling/api-info/definition/operations/DeleteRecordingRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # Rollup rules Source: https://docs.chronosphere.io/control/shaping/shape-metrics/rules/rollup What are rollup rules, and how to create, edit, or delete them. To downsample and aggregate metrics after they're sent by the client but before they're stored, create *rollup rules*. Rollup rules are a type of [aggregation rule](/control/shaping/shape-metrics/rules) that help you reduce the cardinality footprint of your metrics by dropping raw data to eliminate unneeded labels. High cardinality footprints can cause slow dashboards and queries. If you're working with [late-arriving data](/administer/limits-licensing/limits/metric-limits#late-arriving-metrics), rollup rules are well suited for ensuring all of your data aggregates the way you need it. As an example, `instance` or `pod` labels don't often add value on their own, but removing these labels from the client side isn't always possible. You can use rollup rules to avoid storing these labels. Rollup rules support both Prometheus and Graphite metrics. ## View rollup rules Select from the following methods to view rollup rules. In Observability Platform, view rollup rules in the [Aggregation rules UI](/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules). To list only rollup rules on the command line with [Chronoctl](/tooling/chronoctl), use this command: ```shell theme={null} chronoctl rollup-rules list ``` To complete this action with the Chronosphere API, use the [`ListRollupRules`](/tooling/api-info/definition/operations/ListRollupRules) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. For information about viewing, copying, or downloading rule configurations, see [Rule configuration](/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules#rule-configuration). ## Create a rollup rule Select from the following methods to apply rollup rules. Observability Platform doesn't limit the number of rollup rules a system can have. If you define a rollup rule using the Observability Platform app, you must download the rule configuration and apply it with one of the supported methods. Create rollup rule configurations in Observability Platform from the [Aggregation rules UI](/control/shaping/shape-metrics/reduce-cardinality/aggregation-rules). When creating a rule configuration, the **Visual Editor** displays by default. When creating a rule in Metrics Analyzer, the dialog pre-populates fields based on the user's selected data. To create a rule configuration: 1. Enter or edit data for the following fields: * **Rule Name**: Add or edit the name of the rule. * **Rule Details**: Either **Rule Preview** or **Rule Enabled**. See [rule mode](#rule-mode) for more information. * **Matching Time Series:** Time series the rule applies to. You must include a **Label**, operator (`=` or `!=`), and a **Value**. The value you enter maps to the [`filters`](#filters) section of the [CreateRollupRule](/tooling/api-info/definition/operations/CreateRollupRule#body-rollup-rule-filters) endpoint. For example, if you want the rollup rule to match on [Prometheus gauge metrics](/control/shaping/shape-metrics/types#prometheus), enter `__m3_prom_type__` as the label to match on, and `gauge` as the value. The resulting filter looks like: ```text theme={null} __m3_prom_type__ = gauge ``` Values accept a comma-separated list and [glob syntax](/investigate/querying/glob-syntax), including matching multiple patterns with an `OR`, such as `service:{svc1,svc2}`. Click **Add** to add another time series. * **Labels to Roll Up**: **Discard Labels** or **Keep Labels**. Add labels to the **Input Labels** text box. * **Output Metric**: The new metric's name and aggregation configuration. * **Output Metric Name:** Edit the output metric name. Clear the checkbox for **Include metric name** to remove the original name. * **Input Metric Type:** Select a [metric type](/control/shaping/shape-metrics/types#observability-platform-types), which determines how the rollup rule interprets all matching data points. For example, if you select **Gauge**, the rollup rule interprets all matching data points as that data type, even if the original source isn't a gauge metric. This behavior means that the metric type you choose doesn't have to match the data type of the incoming data. If you want to match the incoming metric to a specific type, enter two matching time series in the rollup rule: one to match the metric, and another to match the metric type. Use `__metric_type__` to define the type of metric you want to match on. For example, if you want to match a time series named `agg_write_latency` that's a cumulative exponential histogram, define two series that look like: ```text theme={null} __name__ = agg_write_latency AND __metric_type__ = cumulative_exponential_histogram ``` * **Aggregation:** Select an [aggregation operation](#aggregation-operations). * **Aggregation Interval:** The length of time between samples. See [Aggregation interval](#aggregation-interval). * **Raw Data**: Turn on **Drop raw input data** to remove the raw input data after aggregation. 2. When finished, click **Code Config**. 3. Choose your rule creation method from these options: * **Chronoctl** * **Terraform** * **API** 4. Apply the changes based on your selected method. To create a rollup rule with [Chronoctl](/tooling/chronoctl), define the rule in a YAML file and apply it. If you don't already have a YAML configuration file, use the `scaffold` Chronoctl parameter to generate a template for a specific resource type: ```shell theme={null} chronoctl rollup-rules scaffold ``` You can redirect the results (using the redirection operator `>`) to a file for editing. 1. Create or edit a YAML configuration file to configure the rollup rule. 2. Apply the rollup rule: ```shell /FILE_NAME/ theme={null} chronoctl apply -f FILE_NAME.yaml ``` Replace *`FILE_NAME`* with the name of the YAML configuration file. See the [Chronoctl rollup rule example](#chronoctl-rollup-rule-example) for more information. When you run `terraform plan` to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the [Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs) documentation. Create a rollup rule with Terraform by using the `chronosphere_rollup_rule` type, followed by a name in a resource declaration. See the [Terraform rollup rule example](#terraform-rollup-rule-example) for more information. 1. Add the definition to a Terraform file. 2. Run this command to create the resource: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`CreateRollupRule`](/tooling/api-info/definition/operations/CreateRollupRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. Rollup rules take effect immediately, but can require a full [aggregation interval](#aggregation-interval) to show a change. ### Best practices for rule creation Following these guidelines helps ensure your rollup rules work as intended: * Use [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer) to verify your [glob syntax](/investigate/querying/glob-syntax) to ensure your query matches the correct metrics. * Before using a rollup rule to group labels, be sure those labels aren't used in other places, such as dashboards, monitors, or the queries you use to debug issues. * A dash (`-`) inside square brackets is a range operator, not a literal character. The filter `service_cluster:[a-d]` matches a single character from `a` through `d`. To match a value that contains a literal dash, use curly braces, which take a comma-separated list of alternatives and treat a dash as an ordinary character. For example: `service_cluster:{my-label,other-label}`. * Metrics can match more than one rule. Matching multiple rules can affect data retention. If a rule matches any `drop_raw=true`, raw metrics are dropped. * If a single output series receives more than 10 million unique input series, Observability Platform might stop accepting new input series specified in the rollup rule, which could result in partially aggregated metrics. To avoid this behavior, choose a [label policy](#label-policies) that writes more output series by removing fewer labels. ```mermaid actions={false} theme={null} --- title: Raw data persistence flowchart --- flowchart LR accTitle: Raw data persistence flowchart accDescr: This flowchart shows how metrics matching multiple rules can have unexpected data persistence. R1[Rule 1]-->DRT[drop
raw=true] R2[Rule 2]-->DRT R3[Rule 3]-->DRF[drop
raw=false] ExRule[sample_metric
with
label=test]-->|matches| R1 ExRule-->|matches| R3 ER[End result is
raw data drops
and aggregated
data persists] DRT-->|Raw data
dropped| ER DRF-->|Raw data
persisted| ER ``` ### Chronoctl rollup rule example Here's an example of a rollup rule that matches time series with the value `permits_blocked`, while discarding any labels matching `instance` and `job`. It uses a cumulative counter type metric, and aggregates as a sum using a 30-second interval. ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: RollupRule spec: slug: permits_blocked_without_instance name: permits blocked without instance filters: - name: __name__ value_glob: permits_blocked metric_name: '{{ .MetricName }}:without_instance' metric_type: CUMULATIVE_COUNTER aggregation: SUM interval: 30s label_policy: discard: - instance - job add_metric_type_label: true mode: ENABLED ``` ### Terraform rollup rule example Here's an example of a rollup rule that matches time series with the value `permits_blocked`, while discarding any labels matching `instance` and `job`. It uses a cumulative counter type metric, and aggregates as a sum using a 30-second interval. ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_rollup_rule" "permits_blocked_without_instance" { name = "permits blocked without instance" slug = "permits_blocked_without_instance" filter = "__name__:permits_blocked" metric_type = "CUMULATIVE_COUNTER" aggregation = "SUM" interval = "30s" exclude_by = ["instance", "job"] metric_type_tag = true mode = "ENABLED" new_metric = "{{ .MetricName }}:without_instance" } ``` ## Delete a rollup rule To delete rollup rules with [Chronoctl](/tooling/chronoctl), use this command: ```shell /SLUG/ theme={null} chronoctl rollup-rules delete SLUG ``` Replace *`SLUG`* with the rule's slug. For example, to delete the `http_request_duration_by_service_and_status` rule, use this command: ```shell theme={null} chronoctl rollup-rules delete http_request_duration_by_service_and_status ``` If your slug starts with a dash (`-`), use double quotes (`"`) around the slug name. ```shell theme={null} chronoctl rollup-rules delete "-my-rollup-rule" ``` To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform): 1. Edit your Terraform configuration file to remove the pre-existing resource definition. 2. Run this command to remove the resource from Observability Platform: ```shell theme={null} terraform apply ``` To complete this action with the Chronosphere API, use the [`DeleteRollupRule`](/tooling/api-info/definition/operations/DeleteRollupRule) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Rollup rule attributes To accurately aggregate your data, rollup rules require you to both configure multiple fields and to have an understanding of [aggregation operations](#aggregation-operations). See the [CreateRollupRule API](/tooling/api-info/definition/operations/CreateRollupRule) documentation for the complete list of fields that are part of the `rollup_rule` object that you define when [creating a rollup rule](#create-a-rollup-rule) with any of the supported methods. ### Filters The `filters` field selects the incoming metrics a rule applies to. Each entry matches one label name against a value, and a metric must match every entry to match the rule. Values support [glob syntax](/investigate/querying/glob-syntax), including matching multiple patterns with an `OR`, such as `service:{svc1,svc2}`. Synthetic labels match on request metadata instead of on a metric's label. The following table lists the synthetic labels that a rollup rule filter most often matches on, and the values each one accepts: | Filter | Matches | Valid values | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `__metric_type__` | The [metric type](/control/shaping/shape-metrics/types#observability-platform-types). Use this filter to match on metric type. | `cumulative_counter`, `cumulative_exponential_histogram`, `delta_counter`, `delta_exponential_histogram`, `gauge`, `measurement` | | `__metric_source__` | The [source format](/control/shaping/shape-metrics/types#supported-formats) the metric arrived in. | `carbon`, `chrono_azure`, `chrono_cloudflare`, `chrono_gcp`, `chrono_integration`, `cloudwatch_metric_stream`, `dogstatsd`, `open_metrics`, `open_telemetry`, `prometheus`, `signalfx`, `statsd`, `tagged_statsd`, `wavefront` | | `__m3_prom_type__` | The [Prometheus metric type](/control/shaping/shape-metrics/types#prometheus), for data ingested with Prometheus. | `counter`, `gauge`, `histogram`, `gauge_histogram`, `summary`, `info`, `state_set` | | `__otel_type__` | The [OpenTelemetry metric type](/control/shaping/shape-metrics/types#opentelemetry), for data ingested with OpenTelemetry. | `sum`, `monotonic_sum`, `gauge`, `histogram`, `exp_histogram`, `summary` | For example, the following filter matches any cumulative counter that carries a `service=gateway` label and whose metric name starts with `http_requests_`: ```text theme={null} __metric_type__:cumulative_counter service:gateway __name__:http_requests_* ``` ### Aggregation interval The `interval` field sets the amount of time between the aggregated data points a rule produces. If you omit it, the rule uses the default resolution from your [retention policy](/administer/limits-licensing/licensing). An interval must match a resolution that your tenant can write aggregated data to, typically `15s`, `30s`, or `60s`. ### Rule mode The `mode` field controls whether a rule aggregates data. It accepts two values: * `ENABLED`: Aggregates incoming data according to the rule configuration. A rule that omits `mode` uses this value. * `PREVIEW`: Reports the rule's estimated impact without changing stored data. For the preview workflow, see [Preview a rule's shaping impact](/control/shaping/shape-metrics/reduce-cardinality/shaping-impact). ### Conflicting output metric names Set `skip_on_conflict` to `true` to skip a rule when another rollup rule already produces a metric with the same output name. Use this field to write a rule that produces a series only when no other rule produces one under that name. Default: `false`. ### Label policies Use label policies to define which labels to preserve in the resulting metric. In the rollup rule definition, add the appropriate field to specify which labels to retain or discard. Most rollup rules must set one of these fields, and no rule can set both. Neither field accepts `__name__`. A rollup rule can't group by the metric name or drop it. To change the output metric's name, use `metric_name` instead. Rules that set `graphite_label_policy` can't set `keep` or `discard`, their Terraform equivalents `group_by` and `exclude_by`, or `metric_name`. Manage Graphite positional labels with [`graphite_label_policy.replace`](#set-a-graphite-label-policy) instead. #### Keep specified labels To aggregate only metrics that contain *all* of the specified labels and discard all other labels, use `group_by` (Terraform) or `keep`. When using these rollup rules, you must specify the labels to aggregate the metrics by. If a metric doesn't include all of the specified labels, the metric isn't included in the rule. If a rollup rule uses `group_by` or `keep`, the rule will match only metrics with labels that contain these fields, even if the label `filters` would have matched these metrics. A rule that uses `group_by` or `keep` also constrains its own [filters](#filters). The filters must include a `__name__` entry, and that entry can't contain a wildcard, so the rule targets one metric name instead of an arbitrary set. Adding the `{{ .MetricName }}` template to `metric_name` lifts both restrictions, because the output name then varies with each matched metric. #### Remove specified labels To target a group of metrics for a particular service, team, or other higher-level set of metrics, use `exclude_by` (Terraform) or `discard`. When using these rollup rules, you specify which labels to remove from the aggregated metric, while keeping all other labels. #### Set a Graphite label policy For Graphite metrics, you can use the `graphite_label_policy` parameter to also set a Graphite-specific label policy. This lets you define replacements for label values without changing their positions, which can reduce cardinality without breaking Graphite metrics' preferred positional indexing. For example, assume you have raw metric names that follow this pattern: ```text theme={null} cluster.production.instance.instance1.requests_count cluster.production.instance.instance2.requests_count ... ``` You can create a Graphite label policy that defines a replacement rule that replaces the third positional label name (`__g3__`) with a new string value (`INSTANCE`). This replacement aggregates these metrics as `cluster.production.instance.INSTANCE.requests_count`, without changing their positional indexing. The output of the `chronoctl rollup-rules scaffold` command includes the `graphite_label_policy` parameter: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: RollupRule spec: # ... graphite_label_policy: # List of labels to replace. Use to discard high-cardinality values while still # preserving the original positions of the Graphite metric. replace: - # Name of the label to replace the value for. Only positional Graphite labels # such as '__gX__' are allowed. name: # New value of the replaced label. new_value: # ... ``` To implement the rule from the example scenario as a Chronoctl YAML resource, define the `name` and `new_value` in the list of `replace` values: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: RollupRule spec: # ... graphite_label_policy: replace: - name: "__g3__" new_value: "INSTANCE" # ... ``` Define multiple replacements in a single rollup rule by adding more pairs of `name` and `new_value` to the `replace` list. The `graphite_label_policy` block in the `chronosphere_rollup_rule` resource contains a `replace` block with `name` and `new_value` arguments. For example, to implement the rule from the example scenario as a Terraform resource: ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_rollup_rule" "rollup_rule" { # ... graphite_label_policy { replace { name = "__g3__" new_value = "INSTANCE" } } # ... } ``` ### Aggregation operations Some operations can change the type of the metric during aggregation. The resulting metric type of an aggregation is called the *output* metric type. Even if you are ingesting data with the wrong metric type, configure your rollup rule with the metric type that the ingested data should be. For example, if Chronosphere Observability Platform ingests metrics with type `GAUGE`, but the values actually represent `DELTA_COUNTER`, use a `metric_type=DELTA_COUNTER` rollup rule to aggregate them. Rollup rules support the following aggregation operations: #### `CUMULATIVE_COUNTER` Cumulative counters support these aggregations: * `SUM`: Takes the increase of each individual input series within the configured interval, then sums the increases together according to the configured label policy. The output is the cumulative summed increase across all input series. * `COUNT`: Counts the number of unique input series matched by the configured label policy (for example, cardinality). The output type of all cumulative counter aggregations is a [`CUMULATIVE_COUNTER`](/control/shaping/shape-metrics/types#cumulative-counter). #### `GAUGE` Gauges support the following aggregation methods: * `SUM`: Takes the max value of each individual input series within the configured interval, then sums all final values together by the configured label policy. * `COUNT`: Counts the number of unique input series matched by the configured label policy (for example, cardinality). * `MIN`: Takes the minimum value of all data points within the configured interval across all series matched by the configured label policy. * `MAX`: Takes the maximum value of all data points within the configured interval across all series matched by the configured label policy. * `PXX`, `MEAN`, `MEDIAN`, `STDEV`, `SUMSQ`: Takes the maximum value of each individual input series within the configured interval, and then computes the value distribution. The output type of all gauge aggregations is a [`GAUGE`](/control/shaping/shape-metrics/types#gauge). When querying a gauge metric with a range vector included in the query downsampling might impact the accuracy of the query result. Most use cases that fit this criteria can be converted to use counters instead, which avoids the issue. #### `DELTA_COUNTER` Supported aggregations: * `SUM`: Sums all values of all series matched by the configured label policy. All values must be nonnegative. * `COUNT`: Counts the number of unique input series matched by the configured label policy, such as cardinality. Adding more samples to an existing series doesn't change the count. Only new series impact the count. * `COUNT_SAMPLES`: Counts the number of input samples matched by the configured label policy. Every sample increments the count, even when multiple samples land on the same series. The output type of all delta counter aggregations is a [`DELTA_COUNTER`](/control/shaping/shape-metrics/types#delta-counter). A `DELTA_COUNTER` rule treats three settings as a single group: `metric_name`, `aggregation`, and the label policy, which is exactly one of `keep` or `discard`. Either set all three or omit all three. Setting any one of the three makes the other two required. Omitting all three aggregates matching series as a `SUM`, keeps the original metric name, and retains every label. The effect is to downsample the counter to the rule's interval without otherwise changing the series, which is what you want when you need a coarser resolution and nothing else. `MEASUREMENT` and `DELTA_EXPONENTIAL_HISTOGRAM` rules can also omit `metric_name` and the label policy, but both types require an `aggregation`. Every rollup rule requires a `name`. The `name` field identifies the rule itself, and is distinct from `metric_name`, which names the output metric. #### `MEASUREMENT` A key feature of `MEASUREMENT` aggregations lies in how they treat individual samples. Unlike other types such as [`GAUGE`](/control/shaping/shape-metrics/rules/rollup#gauge) and [`CUMULATIVE_COUNTER`](/control/shaping/shape-metrics/rules/rollup#cumulative_counter), `MEASUREMENT` metrics aggregate all at once, across all samples of your matching time series within the aggregated time interval. This enables calculation of accurate statistics server-side, within Observability Platform. A typical use case for `MEASUREMENT` aggregations is calculating statistics across raw request latencies across all instances. This can be correctly performed through `metric_type=MEASUREMENT` and `aggregation=P95`. Using `metric_type=GAUGE` in this scenario produces results you don't want, discarding all samples except the per-instance max value, then computing the ninety-fifth percentile across these per-instance max values. Every `MEASUREMENT` rule must set `drop_raw` to `true`. Observability Platform rejects a `MEASUREMENT` rule that retains its raw input data. Measurements support the following aggregation methods: * `SUM`: Sums all values of all series matched by the configured label policy. All values must be nonnegative. The output metric type is a [`DELTA_COUNTER`](/control/shaping/shape-metrics/types#delta-counter). * `COUNT_SAMPLES`: Counts the number of input samples matched by the configured label policy. The output metric type is a [`DELTA_COUNTER`](/control/shaping/shape-metrics/types#delta-counter). * `SUMSQ`: Sums the squares of all values of all series matched by the configured label policy. The output metric type is a [`DELTA_COUNTER`](/control/shaping/shape-metrics/types#delta-counter). * `LAST`: Takes the last value of all samples matched by the configured label policy. The output metric type is a [`GAUGE`](/control/shaping/shape-metrics/types#gauge). * `MIN`: Takes the minimum value of all samples matched by the configured label policy. The output metric type is a [`GAUGE`](/control/shaping/shape-metrics/types#gauge). * `MAX`: Takes the maximum value of all samples matched by the configured label policy. The output metric type is a [`GAUGE`](/control/shaping/shape-metrics/types#gauge). * `PXX`, `MEAN`, `MEDIAN`, `STDEV`: Computes the value distribution across all samples matched by the configured label policy. The output metric type is a [`GAUGE`](/control/shaping/shape-metrics/types#gauge). * `HISTOGRAM`: Summarizes the distribution of values as an exponential histogram with a starting scale of 5. The output type is a [`DELTA_EXPONENTIAL_HISTOGRAM`](/control/shaping/shape-metrics/types#delta-exponential-histograms). ### Histograms aggregation operations If either the input histogram or resulting aggregation exceeds the 160-bucket limit, Observability Platform decreases the exponential histogram scale until the bucket count is within the limit. Downscaling reduces the exponential histogram's resolution. #### `CUMULATIVE_EXPONENTIAL_HISTOGRAM` Cumulative exponential histogram aggregations operate on OpenTelemetry exponential histograms with cumulative temporality, and on Prometheus native histograms with an exponential bucket layout. Cumulative exponential histograms support this aggregation method: * `SUM`: Merges input cumulative exponential histograms by the configured label policy. The output metric type is a [`CUMULATIVE_EXPONENTIAL_HISTOGRAM`](/control/shaping/shape-metrics/types#cumulative-exponential-histograms). #### `DELTA_EXPONENTIAL_HISTOGRAM` Delta exponential histogram aggregations operate on OpenTelemetry exponential histograms with delta temporality. Delta exponential histograms support this aggregation method: * `SUM`: Merges input delta exponential histograms by the configured label policy. The output metric type is a [`DELTA_EXPONENTIAL_HISTOGRAM`](/control/shaping/shape-metrics/types#delta-exponential-histograms). # Metric types Source: https://docs.chronosphere.io/control/shaping/shape-metrics/types Learn how counters, gauges, histograms, and other metric types affect aggregation, long-term downsampling, and query results. Each ingested and aggregated metric has a metric type, a critical piece of metadata that indicates what kind of data a metric represents. Metric types affect how the data [aggregates](/control/shaping/shape-metrics/rules/rollup#aggregation-operations) for storage, [long-term downsampling](/control/shaping/shape-metrics/downsampling) behaves, and how that data displays within the Chronosphere Observability Platform. If a metric is ingested or aggregated with the wrong type, you can get unexpected results when aggregating or querying the data, or data loss when persisting data in long term storage. ## Observability Platform types Observability Platform supports ingesting many different metric types. ### Cumulative counter A counter that stores a strictly increasing count of distinct events or a reset-to-zero on restart. Counters are a fundamental metric construct that keeps track of the number of times a certain event has occurred. * Each datapoint stores the running total at the given timestamp. * Aggregated using `metric_type=CUMULATIVE_COUNTER` [rollup rules](/control/shaping/shape-metrics/rules/rollup). * Queryable as a [Prometheus counter](https://prometheus.io/docs/concepts/metric_types/#counter). * Subject to [long-term downsampling](/control/shaping/shape-metrics/downsampling). * View incoming cumulative counters with a `__metric_type__:cumulative_counter` filter in the [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer). Examples of cumulative counters include: * Number of requests served * Number of user logins * Number of heap allocations ### Delta counter A delta counter is a counter where each data point stores a strictly positive delta increment at the given timestamp. For example, a client can send a `1` every time a request is served to count the number of requests served. With this method, the client keeps track only of how many events since the last emission or flush of the metric value, and emits only that value. Unlike a cumulative counter, a delta counter doesn't keep a running sum. * Stores the same kinds of data as cumulative counters, but with a different format. * Aggregated with `metric_type=DELTA_COUNTER` [rollup rules](/control/shaping/shape-metrics/rules/rollup). Raw delta counter data is automatically aggregated unless explicitly dropped by a rollup rule. * Queryable as a [Prometheus counter](https://prometheus.io/docs/concepts/metric_types/#counter). * View incoming delta counters with a `__metric_type__:delta_counter` filter in the [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer). For more information about querying delta counters in Observability Platform, see [Querying delta temporality metrics](/investigate/querying/metrics/delta-queries). ### Gauge A gauge is a basic metric type that tracks a value which changes over time. * Each datapoint stores the value at the given timestamp. It's impossible to have two values at the same point in time. * Aggregated with `metric_type=GAUGE` [rollup rules](/control/shaping/shape-metrics/rules/rollup). * Queryable as a [Prometheus gauge](https://prometheus.io/docs/concepts/metric_types/#gauge). * Subject to [long-term downsampling](/control/shaping/shape-metrics/downsampling). * View incoming gauges with a `__metric_type__:gauge` filter in the [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer). Some examples of gauges are: * Current number of in-flight requests. * Current number of signed-in users. * Current amount of memory in use. ### Histogram The frequency of values in a set of measurements that fall within certain ranges called *buckets*. A histogram uses several buckets defined in a *bucket layout*. Observability Platform processes and persists each histogram as a single structured value and time series. Observability Platform supports OpenTelemetry exponential histograms and Prometheus native histograms, and also supports both cumulative and delta temporality. The histogram's bucket layout defines the histogram metric type in Observability Platform. You can query histograms using [PromQL histogram functions](https://prometheus.io/docs/prometheus/latest/querying/functions/). They're subject to the same long-term downsampling as other metric types. Observability Platform processes and persists Prometheus histograms as individual cumulative counter or delta counter time series. To query both histograms and classic Prometheus histograms, see [Querying histograms](/investigate/querying/promql/apply-functions#querying-histograms). Some examples of histograms include: * Distributions of request latencies * Distributions of payload sizes | Metric Type | Bucket Layout | Temporality | Description | | -------------------------------- | ------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Cumulative exponential histogram | Exponential | Cumulative | A cumulative histogram with exponential bucket layout summarizes a set of measurements by defining non-overlapping, exponentially increasing bucket ranges and counting the frequency of observations within each bucket range. It includes the total count and sum of all observations. See [Cumulative exponential histograms](#cumulative-exponential-histograms). | | Delta exponential histogram | Exponential | Delta | A delta exponential histogram has the same properties of a cumulative exponential histogram. The only difference is the delta temporality, where the instrumentation client only summarizes and sends observations since the last emission or flush of the histogram metric value. See [Delta exponential histograms](#delta-exponential-histograms). | Histograms are limited in these ways: * The maximum bucket limit is 160 buckets. * The maximum exponential histogram scale is 8, and the minimum scale is -4. #### Cumulative exponential histograms The cumulative exponential histogram type is compatible with both OpenTelemetry cumulative exponential histograms and classic Prometheus native histograms with exponential bucket layouts. * Observability Platform automatically reduces the exponential histogram scale when the histogram exceeds the 160-bucket limit. When downscaling, it doubles the exponential histogram bucket width to summarize the observations with fewer but wider buckets, effectively reducing the exponential histogram's resolution. When aggregating or querying exponential histograms with different bucket scales, Observability Platform scales them down to the lowest common histogram scale. Zero-threshold handling similarly widens scales until values fit. * Observability Platform aggregates them with `metric_type=cumulative_exponential_histogram` rollup rules. * You can view incoming measurements by using a `__metric_type__:cumulative_exponential_histogram` filter in the Live Telemetry Analyzer. #### Delta exponential histograms The delta exponential histogram type is compatible only with OpenTelemetry delta exponential histograms. * Observability Platform aggregates them with `metric_type=delta_exponential_histogram` rollup rules. * You can view incoming measurements by using a `__metric_type__:delta_exponential_histogram` filter in the Live Telemetry Analyzer. For more information about querying delta histograms in Observability Platform, see [Querying delta temporality metrics](/investigate/querying/metrics/delta-queries). ### Measurement A raw observation with some discrete value. Measurements differ from counters as each measurement event has a discrete, arbitrary value. Measurements also differ from a gauge, as there can be multiple values at any point in time. * You must use a `metric_type=MEASUREMENT` [rollup rule](/control/shaping/shape-metrics/rules/rollup) with `drop_raw=true` to aggregate measurements into a persistable metric type. Raw measurement data can't be persisted. * The `MEASUREMENT` metric type is best used when all values should be [aggregated](/control/shaping/shape-metrics/rules/rollup#measurement) at once across many matching time series. * View incoming measurements with a `__metric_type__:measurement` filter in the [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer). Some examples of measurements are: * The latency of an individual request * The payload size of an individual request ## Supported formats Observability Platform can ingest metrics from many different [external metric clients](/ingest/metrics-traces/collector/addl-metrics). Each external metric client has their own concept of "metric type" which is mapped into a Observability Platform metric type. ### Prometheus | Prometheus Type | Observability Platform Type | | ----------------------------------------------- | -------------------------------- | | Gauge | Gauge | | Counter | Cumulative counter | | Histogram | Cumulative counter | | Summary (quantile data points) | Gauge | | Summary (sum and count data points) | Cumulative counter | | Native histogram with exponential bucket layout | Cumulative exponential histogram | ### OpenTelemetry | OpenTelemetry Type | Observability Platform Type | | ----------------------------------- | -------------------------------- | | Gauge | Gauge | | Cumulative monotonic sum | Cumulative counter | | Delta monotonic sum | Delta counter | | Cumulative histogram | Cumulative counter | | Delta histogram | Delta counter | | Summary (quantile data points) | Gauge | | Summary (sum and count data points) | Cumulative counter | | Cumulative non-monotonic sum | Gauge | | Cumulative exponential histogram | Cumulative exponential histogram | | Delta exponential histogram | Delta exponential histogram | Observability Platform doesn't support delta non-monotonic sums (also known as delta up-down counters). ### Datadog | Datadog Type | Observability Platform Type | | ------------ | --------------------------- | | Gauge | Gauge | | Count | Delta counter | | Histogram | Measurement | | Distribution | Measurement | ### SignalFX | SignalFX Type | Observability Platform Type | | ------------- | --------------------------- | | Gauge | Gauge | | Cumulative | Cumulative counter | | Counter | Delta counter | ### Wavefront | Wavefront Type | Observability Platform Type | | -------------- | --------------------------- | | Gauge | Gauge | | Counter | Cumulative counter | | Delta counter | Delta counter | | Histogram | Cumulative counter | ### StatsD | StatsD Type | Observability Platform Type | | ----------- | --------------------------- | | Gauge | Gauge | | Counter | Delta counter | | Timer | Measurement | ### Google Cloud | Google Cloud Type | Observability Platform Type | | ----------------- | --------------------------- | | Gauge | Gauge | | Cumulative | Cumulative counter | | Delta | Delta counter | | Distribution | Delta counter | Google Cloud distributions are transformed to [Prometheus classic histograms](/investigate/querying/promql/apply-functions#querying-classic-prometheus-histograms). They're stored as separate delta counter time series per distribution bucket, with a separate time series for the distribution count and sum. # Data control concepts Source: https://docs.chronosphere.io/control/storage Learn about data control concepts in Observability Platform and how you can use them to control your data. Chronosphere uses [collectors](/ingest) to ingest data, and utilizes push and pull models of ingestion, depending on the telemetry data collected and the method of ingestion. Chronosphere provides the following control concepts to help manage your data and keep only the data that's most important to your organization. ## Consumption concepts In a consumption model, you create *partitions*, which are slices of your data that have distinct owners. Partitions provide a consistent structure for attributing usage and costs to the appropriate owners in your organization so they can isolate and control independent parts of the business. Partitions let you apply *budgets*, which are optional shaping policies you can assign to a partition to safeguard against runaway usage and overspending. Budgets provide flexibility to enforce accountability at the correct level of ownership. Budgets include thresholds that define which actions to take when a threshold is exceeded. To learn more about these control mechanisms, see [Manage consumption](/control/consumption). ## Capacity concepts The capacity model is being replaced by the [consumption model](#consumption-concepts). In a capacity model, metrics use *quotas* to assign specific percentages of your total persisted writes limit to *pools* of metrics. Each pool might represent teams or other logical groupings within your organization. Use metric quotas to give each pool a specific quota of your total persisted writes license, expressed either as a percentage or a value in data points per second (DPPS). To learn more about these control mechanisms, see [quotas and pools](/control/shaping/shape-metrics/quotas). ## Trace control concepts In both the consumption and capacity models, you can use *datasets* as a proactive control mechanism to organize your data and map it to named groups relevant to your organization. Datasets can either overlap or stand alone, and can be assigned to only a single business unit or service within your organization. Each dataset entity lets you define a filter to assign specific chunks of data to a particular dataset. A default, system-defined dataset exists for budgeting. You can assign the default dataset to data that isn't explicitly assigned to any other dataset. To change the sampling rates of one or more datasets and to more effectively control your persisted data, use *behaviors*. This shaping mechanism lets you change sampling rates of one or more datasets without needing to write fine-grained sampling or shaping rules. To learn more about these control mechanisms, see [sample your traces](/control/shaping/sample-traces). # Early access program Source: https://docs.chronosphere.io/early-access Learn about the Chronosphere early access program and what it means to participate. Chronosphere offers an Early Access (EA) program, which gives select customers the opportunity to preview and influence upcoming features before they reach General Availability (GA). Your participation is essential in helping Chronosphere build exceptional products. ## Program benefits The EA program offers the following benefits: * **First access to new features**: Try out new capabilities weeks or months before GA release, allowing your teams to experiment with new features, plan implementations, and train users. * **Influence product direction**: Your feedback directly shapes feature design and capabilities. EA participants work closely with Chronosphere product teams to ensure features make observability easier and more intuitive, and integrate seamlessly into your workflows. * **Strategic partnership**: Gain visibility into the Chronosphere product roadmap and build stronger relationships with product and engineering teams through dedicated feedback channels. ## Considerations EA features are under active development. Changes will occur without advanced notification, and can range from cosmetic updates to modifications that require significant workflow changes. This iterative approach lets Chronosphere rapidly incorporate customer feedback and refine features before GA. Maintenance and support for EA features operate differently than GA features. Although Chronosphere makes a best effort to quick address major bugs or issues during the EA period, EA features aren't subject to the 99.9% uptime obligation promised in the SLA for GA features. To protect your operations, Chronosphere won't release any features to the EA program that involve billing changes or have potential data quality impacts. These features undergo additional validation and are available only in GA. # Chronosphere documentation Source: https://docs.chronosphere.io/index Learn about the aspects of the Chronosphere documentation. Select your Chronosphere product to view its product documentation. ## Chronosphere Observability Platform A SaaS-based observability and monitoring platform made for cloud-native infrastructures. Monitor telemetry data across your entire system to identify and solve issues. Minimize impact and downtime for your app and your users while controlling what data you pay for. ## Chronosphere Telemetry Pipeline An observability pipeline platform for routing and processing telemetry data. Integrate with your existing observability tools to send logs, metrics, and traces from any source to any destination. Connect to these sources and destinations through a variety of built-in integration plugins. # Ingest telemetry data Source: https://docs.chronosphere.io/ingest Use control mechanisms to manage the telemetry data you ingest into Chronosphere Observability Platform. Before you can use Chronosphere Observability Platform to view and manage your telemetry data (of whatever type), you need to get that data into Observability Platform. ```mermaid actions={false} theme={null} flowchart LR accTitle: Chronosphere Observability Platform architecture diagram accDescr: Architecture diagram that shows the flow of data from a customer's environment into the Control Plane in the Chronosphere Observability Platform, and then out to third party incident management and alerting software. subgraph "Chronosphere" cp((Chronosphere
Observability Platform)) end subgraph "Your environment" inf(Infrastructure
applications) -- Metrics and
traces --> coll(Chronosphere
Collector) inf -- Logs and
change events --> cp inf -- Logs --> pipe(Chronosphere
Telemetry Pipeline
or existing pipeline) inf -- Logs, metrics,
and traces --> otel(OpenTelemetry
Collector) pipe --> cp coll --> cp otel --> cp end click coll "/ingest/metrics-traces/collector" click pipe "/ingest/pipeline/v2" click otel "/ingest/otel-ingestion" classDef platform fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF; class cp,coll,pipe,otel platform ``` ## Supported ingestion methods Observability Platform supports multiple methods to ingest telemetry data, which depend on which type of data you want to ingest: | Ingestion method | Change events | Logs | Metrics | Traces | | ---------------------------------------------------------- | :--------------------------: | :--------------------------: | :--------------------------: | :--------------------------: | | [Chronosphere Collector](/ingest/metrics-traces/collector) | | | | | | [OpenTelemetry](/ingest/otel-ingestion) | | | | | | [Telemetry Pipeline](/ingest/pipeline/v2) | | | | | | [Existing pipeline](/ingest/logs) | | | | | | [Direct to Observability Platform](/ingest/third-party) | | | | | After ingesting telemetry data, you can use the control mechanisms that Observability Platform provides to [control costs](/control) and ensure you're ingesting only the data you care about. ### OpenTelemetry support For more information about using OpenTelemetry to ingest logs, metrics, and traces, see [OpenTelemetry support in Observability Platform](/ingest/otel-ingestion). ## Ingestion models Chronosphere utilizes *push* and *pull* models of ingestion, depending on the data collected and the method of ingestion. | Telemetry type | Push | Pull | | -------------- | :--------------------------: | :--------------------------: | | Metrics | | | | Traces | | | | Logs | | | *Pull* models, like the Chronosphere Collector, scrape telemetry data from external sources and pull it in. These metrics have consistent reporting intervals. *Push* models, like tracing, send telemetry data to Chronosphere, which is then processed. These metrics can have a broad spectrum of reporting frequency, from large bursts of data to long periods with no data reporting. The ingestion model depends on the telemetry data source. Metrics pushed to Observability Platform can have [latency delays](/ingest/metrics-traces/gcp#metrics-availability) or [sparse time series](/investigate/querying/metrics/troubleshooting#sparse-time-series), which can result in unexpected query results. # Ingest log data Source: https://docs.chronosphere.io/ingest/logs Learn about how to ingest log data into Observability Platform. Ingest data from any of the following supported sources into Chronosphere Observability Platform.

Configure an HTTP output plugin to route logs from Fluent Bit.

Use the Google Cloud Pub/Sub to route logs from Google Cloud Platform.

Use the logging HTTP endpoint to send logs directly to Chronosphere.

Configure an HTTP output plugin to route logs from Logstash.

Use the OpenTelemetry protocol (OTLP) HTTP endpoint to route logs from OpenTelemetry.

Use Chronosphere Telemetry Pipeline to route logs from multiple sources.

 After ingesting log data, verify Observability Platform is receiving your logs as anticipated.

Verify that Observability Platform is receiving your log data.

 # Route logs from Fluent Bit Source: https://docs.chronosphere.io/ingest/logs/fluentbit-logs Learn how to route logs from Fluent Bit to Chronosphere Observability Platform. You can use Fluent Bit to send logs to Chronosphere Observability Platform. To do so, add an [HTTP output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/http) that specifies your Observability Platform tenant as its destination. ## Example configuration To route logs from Fluent Bit to Observability Platform, Chronosphere recommends configuring an HTTP output plugin with the following parameters and values: ```yaml theme={null} pipeline: # ... outputs: - name: http match: "kube.*" alias: Chronosphere host: TENANT port: 443 uri: /api/v1/data/logs/ingest header: "API-Token ${API_TOKEN}" format: json compress: gzip json_date_format: iso8601 json_date_key: TIME tls: true tls.verify: true tls.debug: 1 net.keepalive: true net.keepalive_idle_timeout: 30s net.connect_timeout: 10s net.connect_timeout_log_error: true net.keepalive_max_recycle: 2000 ``` ```ini theme={null} [OUTPUT] Name http Match kube.* Alias Chronosphere Host TENANT Port 443 URI /api/v1/data/logs/ingest Header API-Token ${API_TOKEN} Format json Compress gzip json_date_format iso8601 json_date_key TIME tls On tls.verify On tls.debug 1 net.keepalive On net.keepalive_idle_timeout 30s net.connect_timeout 10s net.connect_timeout_log_error On net.keepalive_max_recycle 2000 ``` * The service account must have read access to route log data to Observability Platform. * * Replace *`TIME`* with the value you assigned to timestamps in your [log normalization settings](/control/shaping/shape-logs/normalize-logs). If you didn't set up log normalization for timestamps, replace *`TIME`* with `timestamp`. # Route logs from Google Cloud Platform Source: https://docs.chronosphere.io/ingest/logs/gcp-logs Learn how to route logs from Google Cloud Platform into Chronosphere Observability Platform. Google Cloud Platform provides [sinks](https://cloud.google.com/logging/docs/routing/overview#sinks), which control how you can route log data to supported destinations. Use the [Google Cloud Pub/Sub](https://cloud.google.com/pubsub/docs) destination to route log data from Google Cloud Platform to Chronosphere Observability Platform. Complete the following steps to route logs from Google Cloud Platform: 1. [Create a sink for logs](#create-a-sink-for-logs). 2. [Create a service account](#create-a-service-account). 3. [Create a pull subscription](#create-a-pull-subscription). 4. [Contact Chronosphere Support](#contact-chronosphere-support). If you want to use Terraform to manage routing Google logs to Observability Platform, refer to the [Terraform example](#terraform-example). After applying this configuration, Terraform generates the information that you supply to [Chronosphere Support](#contact-chronosphere-support). This integration is only intended to handle throughput with low volume. To determine whether this integration is suited to your use case, contact [Chronosphere Support](/support). If you want to route metrics data from Google Cloud Platform, see [Ingest Google Cloud metrics](/ingest/metrics-traces/gcp). ## Create a sink for logs First, you need to create a sink that defines the service type and destination to route your logs. 1. In the Google Cloud Logs Console, in the left navigation, click **Log Router** to open the [Log router](https://console.cloud.google.com/logs/router) page. 2. Follow the steps in the Google Cloud documentation to [create a sink](https://cloud.google.com/logging/docs/export/configure_export_v2#creating_sink). 3. Select **Cloud Pub/Sub topic** as the sink service. 4. Optional: Enter a filter expression that matches the log entries you want to include. See the [Logging query language](https://cloud.google.com/logging/docs/view/logging-query-language) in the Google Cloud documentation for information about how to construct queries. 5. Complete the remaining steps to create the sink. Next, [create a service account](#create-a-service-account). ## Create a service account After creating a sink, you create a service account in Google Cloud that Observability Platform impersonates. Before starting this process, obtain the Observability Platform principal for Google Cloud. Click your [profile icon](/navigate#your-account) from the menu bar and select **My Account**. Copy the value next to **Google Cloud Logs IAM Principal**. Each Google Cloud service account must grant access to the Observability Platform principal to impersonate them. 1. In the Google Cloud console, [create a service account](https://cloud.google.com/iam/docs/service-accounts-create#creating). The service account must have read access to route log data to Observability Platform. 2. Add the Observability Platform principal you obtained previously to the Google Cloud service account. 3. Grant the principal the `iam.serviceAccountTokenCreator` role. 4. Enter information for the remaining fields to finish creating the service account. Next, [create a pull subscription](#create-a-pull-subscription). ## Create a pull subscription Add a [pull subscription](https://cloud.google.com/pubsub/docs/subscription-overview) to the Pub/Sub destination you previously created. Observability Platform initiates requests to the Pub/Sub server to retrieve messages. 1. In the Google Cloud console, [Create a pull subscription](https://cloud.google.com/pubsub/docs/create-subscription#create_a_pull_subscription). 2. In the **Delivery type** menu, select **Pull**. 3. Configure the following recommended settings: | Setting | Value | | -------------------------- | ------------------------------------- | | Message retention duration | 23 hours\* | | Expiration period | Never expire | | Acknowledgement deadline | 60 seconds | | Exactly once delivery | No | | Message ordering | No | | Dead lettering | No | | Retry policy | Retry after exponential backoff delay | \*Set **Message retention duration** to less than 24 hours to avoid storage costs. 1. Grant access to the Pub/Sub you created by assigning the following roles to your new internal service account: * `roles/pubsub.subscriber` * `roles/pubsub.viewer` Lastly, [contact Chronosphere](#contact-chronosphere-support) and provide the details about your pull subscription. ## Contact Chronosphere Support After completing the previous steps, [contact Chronosphere Support](/support) and provide the following information: * The name of your new Google Cloud service account email, such as: ```text theme={null} SERVICE-ACCOUNT@PROJECT-ID.iam.gserviceaccount.com ``` * *`SERVICE-ACCOUNT`* is the name of the Google Cloud service account you created in [create a service account](#create-a-service-account). * *`PROJECT-ID`* is the ID of your Google Cloud project. * The name of the logging sink subscription you created, such as ```text theme={null} projects/PROJECT-ID/subscriptions/SUBSCRIPTION ``` * *`SUBSCRIPTION`* is the name of the pull subscription you created in [create a pull subscription](#create-a-pull-subscription). Chronosphere can then enable the logging integration to start routing your Google Cloud logs to Observability Platform. ## Terraform example The following code provides an example for creating a single Google Cloud service account in the a Google Cloud project, and enables Observability Platform to impersonate and gain access. ```terraform expandable Terraform example icon="square-terminal" theme={null} locals { // Email address of your Chronosphere Logs tenant-specific principal. chronosphere_sa_email = "gcp-logs-TENANT@chronosphere-production-b.iam.gserviceaccount.com" // Google Cloud project containing logging data to be ingested into // Observability Platform. logging_project_id = "PROJECT_ID" // Organization ID of the Google Cloud organization containing the project. org_id = "ORGANIZATION_ID" } // Creates Pub/Sub topic for logging sink. resource "google_pubsub_topic" "main" { name = "logging_sink" project = local.project_id } // Creates a new logging sink. resource "google_logging_organization_sink" "main" { org_id = local.org_id name = "all_logs_sink" destination = "pubsub.googleapis.com/${google_pubsub_topic.main.id}" include_children = true } // Grants sink permission to roles so they can write to the Pub/Sub topic. data "google_iam_policy" "topic" { binding { role = "roles/pubsub.publisher" members = [ google_logging_organization_sink.main.writer_identity, ] } } // Binds the role to a new topic. resource "google_pubsub_topic_iam_policy" "main" { project = local.project_id topic = google_pubsub_topic.main.name policy_data = data.google_iam_policy.topic.policy_data } // Creates a new Pub/Sub subscription. resource "google_pubsub_subscription" "main" { name = "all_logs_sink" project = local.project_id topic = google_pubsub_topic.main.name message_retention_duration = "23h" ack_deadline_seconds = 60 retry_policy { minimum_backoff = "10s" } enable_message_ordering = false enable_exactly_once_delivery = false } // Creates a policy granting the new Chronosphere logs service account access to // the subscription. data "google_iam_policy" "subscription" { binding { role = "roles/pubsub.viewer" members = [ google_service_account.chronosphere_logs.member ] } binding { role = "roles/pubsub.subscriber" members = [ google_service_account.chronosphere_logs.member ] } } // Binds the new policy to the subscription. resource "google_pubsub_subscription_iam_policy" "main" { project = local.project_id subscription = google_pubsub_subscription.main.name policy_data = data.google_iam_policy.subscription.policy_data } // Service account that lets an Observability Platform tenant-specific principal // to impersonate it. resource "google_service_account" "chronosphere_logs" { project = local.project_id account_id = "chronosphere-logs" } // The service account provides the Observability Platform tenant-specific principal with // roles/iam.serviceAccountTokenCreator access so that it can impersonate it. Only // the Observability Platform tenant-specific principal can perform this // impersonation. data "google_iam_policy" "chronosphere_logs" { binding { role = "roles/iam.serviceAccountTokenCreator" members = ["serviceAccount:${local.chronosphere_sa_email}"] } } // Assigns the token creator permission to the service account. resource "google_service_account_iam_policy" "chronosphere_logs" { service_account_id = google_service_account.chronosphere_logs.name policy_data = data.google_iam_policy.chronosphere_logs.policy_data } ``` # Route logs over HTTP Source: https://docs.chronosphere.io/ingest/logs/http-logs Use the Chronosphere logging HTTP endpoint to send logs to your Observability Platform tenant over HTTP. Use the Chronosphere logging HTTP endpoint to send your logs to Observability Platform over HTTP. The endpoint uses the following format: ```text theme={null} https://TENANT.chronosphere.io/api/v1/data/logs/ingest ``` * In addition to formatted log data, such as JSON, the logging HTTP endpoint accepts logs in plain text format. To minimize egress costs, this endpoint supports `gzip`, `snappy`, and `zstd` compression methods. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. The service account must have read access to route log data to Observability Platform. The following example shows a formatted `curl` request that includes the logging HTTP endpoint: ```shell wrap theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}.chronosphere.io/api/v1/data/logs/ingest" ``` ## Limits The logging HTTP endpoint has the following limits: * Logs with timestamps exceeding 24 hours into the future or past are rejected. * Requests (compressed) exceeding 50 MB are rejected. ## Status codes The logging HTTP endpoint returns the following status codes: * `200`: Accepted. The request was accepted for processing. * `413`: Large request. The maximum content size per payload (compressed) is 50 MB. # Route logs from Logstash Source: https://docs.chronosphere.io/ingest/logs/logstash-logs Learn how to route logs from Logstash to Chronosphere Observability Platform. You can route log data from Logstash to Chronosphere Observability Platform. To route logs, configure an [HTTP output plugin](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html) in your [Logstash pipeline configuration file](https://www.elastic.co/guide/en/logstash/current/config-setting-files.html) that specifies your Observability Platform tenant as a destination. 1. In your Logstash pipeline configuration file, add an `output` section that defines the `http` plugin: ```text theme={null} output { http { http_method=>"post" headers => { "Content-Type" => "application/json" "API-token" => "API_TOKEN" } url=>"https://TENANT.chronosphere.io/api/v1/data/logs/ingest" format=>"json_batch" } } ``` * The service account must have read access to route log data to Observability Platform. * * Optional: The `format=>"json_batch"` option collects each batch of events received by the output and places them into a single JSON array that's sent in one request. 2. After defining the output, [contact Chronosphere Support](/support) and indicate which field in your data contains log timestamps. # Use OpenTelemetry protocol endpoints Source: https://docs.chronosphere.io/ingest/logs/otel-logs Learn how to configure and use Chronosphere Observability Platform endpoints for OpenTelemetry protocol ingestion of log data. Chronosphere Observability Platform supports ingesting log data using OpenTelemetry protocol (OTLP) HTTP ingestion endpoints. For best results, ingest telemetry through a local OpenTelemetry Collector for processing and batching. Sending telemetry directly from an OpenTelemetry SDK can be less reliable. ## Requirements Your OTLP configuration must meet certain requirements. * **`API-Token` header:** All requests must include the `API-Token` HTTP header, and its value must be the API token of a valid [service account](/administer/accounts-teams/service-accounts) with write permissions. * **OTLP version support:** Only v1.0 and higher are supported. * **Supported compression methods:** Chronosphere supports `gzip`, `snappy`, and `zstd` compression, and recommends `zstd`. * **Encryption:** All requests must be encrypted using TLS. ### Recommendations Chronosphere recommends setting the timeout value for a logs request to 30 seconds, and to enable retries to prevent transient network errors from causing data loss. ## OTLP endpoint URLs Observability Platform provides HTTP endpoints for logs, metrics, and traces. | Telemetry type | HTTP endpoint URL | | -------------- | -------------------------------------------------------------- | | Logs | `https://TENANT.chronosphere.io/data/opentelemetry/v1/logs` | | Metrics | `https://TENANT.chronosphere.io/data/opentelemetry/v1/metrics` | | Traces | `https://TENANT.chronosphere.io/data/opentelemetry/v1/traces` | OpenTelemetry exporters complete the path based on the telemetry type. For the `otlphttp` exporter, set the endpoint base path as follows: ```yaml theme={null} exporters: otlphttp/chronosphere: endpoint: https://TENANT.chronosphere.io/data/opentelemetry/ compression: zstd headers: API-Token: ${env:API_TOKEN} # ... ``` For an example OTLP exporter configuration, see [Configure the OpenTelemetry Collector](/ingest/metrics-traces/otel/otel-ingest#configure-the-opentelemetry-collector). # Route logs through Telemetry Pipeline Source: https://docs.chronosphere.io/ingest/logs/pipeline-logs Learn how to route log data through Chronosphere Telemetry Pipeline to Chronosphere Observability Platform. You can use Chronosphere Telemetry Pipeline to ingest logs from various sources such as Fluent Bit, Open Telemetry, and HTTP, apply parsers and processing rules, and send the output to Chronosphere Observability Platform. You create and define a pipeline in Telemetry Pipeline and set Observability Platform as a destination. After deploying your pipeline, processed log data streams to Observability Platform and is available for [exploring and querying](/investigate/querying/query-logs) in Logs Explorer. ## Prerequisites Before creating a pipeline, you must [install Telemetry Pipeline](/ingest/pipeline/v2/install), which includes installing a Core Operator and Core Instance. ## Create a pipeline Specify a source and destination, and apply parsers and processing rules. To route logs through Telemetry Pipeline to Observability Platform: 1. [Create a pipeline](/ingest/pipeline/v2/build/create-modify#create-a-pipeline). Follow the steps outlined in the Telemetry Pipeline documentation. 2. Add a source, such as Fluent Bit, Elasticsearch, or OpenTelemetry. 3. Add the [Chronosphere Logs](/ingest/pipeline/plugins/destination-plugins/chronosphere) destination. 4. Click the **Chronosphere Logs** destination to edit its configuration: 1. In the **General** section, in the **Host** field, enter: ```text /TENANT/ theme={null} TENANT.chronosphere.io ``` 2. Expand the **Advanced** section, and in the **URI** field, enter: ```text theme={null} /api/v1/data/logs/ingest ``` 5. Complete the remaining steps to deploy your pipeline. After deploying your pipeline, [verify that Observability Platform is receiving your logs](/ingest/logs/verify-logs) as anticipated. # Verify Observability Platform is receiving log data Source: https://docs.chronosphere.io/ingest/logs/verify-logs Learn how to verify that Chronosphere Observability Platform is ingesting your log data. After [ingesting data](/ingest/logs), use Logs Explorer to verify that Chronosphere Observability Platform is receiving log data. 1. In the navigation menu select ** Explorers > Logs Explorer**. 2. On the **Logs Explorer** page, select a time window that includes the period when Observability Platform started ingesting log data. The timeline graph displays a spike at the time when Observability Platform started ingesting log data. 3. Verify the `message`, `timestamp`, `service`, and `severity` fields are being assigned correctly. If these fields aren't mapped in the way you intended, contact [Chronosphere Support](/support). # Ingest metric and trace data Source: https://docs.chronosphere.io/ingest/metrics-traces Learn about the collection methods used to ingest metrics and traces into Chronosphere Observability Platform. Chronosphere Observability Platform supports two [Collectors](#chronosphere-collector-or-opentelemetry-collector) to ingest metric and trace data, the Chronosphere Collector and the OpenTelemetry Collector. Observability Platform also supports direct metric imports from [Google Cloud](/ingest/metrics-traces/gcp). ## Chronosphere Collector or OpenTelemetry Collector Your Collector selection depends on the telemetry data you plan to ingest. Chronosphere recommends using the vendor-supported, optimized, and highly performant [Chronosphere Collector](/ingest/metrics-traces/collector) for Prometheus discovery and scrape workloads. This use case typically works best with metrics data. Use the [OpenTelemetry Collector](/ingest/metrics-traces/otel) if your organization uses OpenTelemetry SDKs for app instrumentation. An added benefit of using the OpenTelemetry Collector is you can configure dynamic, remotely configurable [head sampling](/control/shaping/sample-traces/head-sampling), which is a powerful control mechanism for managing your tracing costs. Observability Platform supports the [OpenTelemetry JaegerRemoteSampler](https://opentelemetry.io/docs/specs/otel/trace/sdk/#jaegerremotesampler) head sampling standard for remotely controlling the sampling configuration of OpenTelemetry SDKs. | Chronosphere Collector | OpenTelemetry Collector | | :-------------------------------------- | :----------------------------- | | Proprietary | Open source | | Fully supported by Chronosphere Support | Exporter configuration support | | No remote trace sampling support | Remote trace sampling support | Chronosphere doesn't provide customer support for OpenTelemetry components. ## Integrations Observability Platform supports these metrics integrations:

Configure AWS to stream metrics to Observability Platform through CloudWatch Metrics Streams.

Connect Observability Platform with Azure Monitor to ingest metrics.

Connect Observability Platform with Google Cloud to ingest metrics from Google Cloud projects.

Use the OpenTelemetry protocol (OTLP) HTTP endpoint to route metrics from OpenTelemetry.

 # Ingest AWS CloudWatch metrics Source: https://docs.chronosphere.io/ingest/metrics-traces/aws-cloudwatch Send AWS CloudWatch metrics to Observability Platform Chronosphere Observability Platform supports receiving AWS CloudWatch metrics through [CloudWatch Metrics Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html). You can configure Amazon Web Services (AWS) to continually stream metrics to Observability Platform by configuring CloudWatch Metric Streams, either manually in the AWS Management Console or by using Terraform. ## Metric naming conventions Metric names of ingested CloudWatch metrics in Observability Platform follow the prefix naming pattern: ```text theme={null} __ ``` * ``: The namespace is lowercased, and Observability Platform replaces all forward slash (`/`) and period (`.`) characters in the CloudWatch namespace with underscores (`_`). All AWS service namespaces follow the naming convention `AWS/`, where `` is replaced with the service name. In Observability Platform, the ingested metrics therefore begin with `aws_`. For a list of AWS services and their respective namespaces, see [AWS services that publish CloudWatch metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html). If you create custom metrics, the namespace you set for the metric correspondingly becomes the metric name prefix in Observability Platform. * ``: Observability Platform preserves the CloudWatch metric name's case. * ``: Observability Platform appends the CloudWatch statistic's name (`count`, `sum`, `maximum`, `minimum`, `average`). If you define additional statistics for a metric, Observability Platform appends the corresponding CloudWatch metric statistic name (`pXX`). For examples, see [Example metric names](#example-metric-names). ## Metric labeling conventions Observability Platform adds CloudWatch metric dimensions as labels to the time series following the pattern `dimension_`. For examples, see [Example metric names](#example-metric-names). ## Metric limitations Most AWS services publish metrics to CloudWatch in or near real-time. However, a subset of AWS services, such as Amazon S3 daily storage metrics for buckets in CloudWatch, produce metrics based on a daily schedule with a data point timestamp older than the two-hour late-arriving data point limit. Observability Platform generally expects data points to be timestamped at the time they are sent, and enforces a two-hour age limit on ingested data points. Check the send interval for a specific AWS service metric to ensure it publishes metrics more frequently than the two-hour ingestion age limit. ## Configure CloudWatch Metric Streams The following diagram shows the architecture and data flow from your AWS account to Chronosphere. In each of your AWS regions where you want to stream data from, a CloudWatch Metric Streams instance sends data to an AWS Data Firehose, which forwards that data to the AWS Data Firehose ingest endpoint running in your Observability Platform tenant. Observability Platform processes the CloudWatch metrics and makes them available for use in queries, monitors, and dashboards. ```mermaid actions={false} theme={null} graph LR subgraph AWS-account subgraph US-EAST-2 cms1[CloudWatch Metrics Stream] adf1[AWS Data Firehose] end subgraph US-WEST-2 cms2[CloudWatch Metrics Stream] adf2[AWS Data Firehose] end cms1 --> adf1 cms2 --> adf2 end subgraph op[Observability Platform] subgraph Tenant adf3[AWS Data Firehose ingest API] ccp[Chronosphere Control Plane] end end adf3 --> ccp adf1 & adf2 --> adf3 classDef greenFill stroke:#60E684,fill:#2FBF71,stroke-width:2px,color:#FFFFFF class adf3,ccp greenFill; classDef greenFillDark stroke:#60E684,fill:#2FBF71,stroke-width:2px,color:#FFFFFF class op greenFillDark; classDef blueFill fill:#000243,stroke-width:2px,color:#FFFFFF class adf1,adf2,cms1,cms2,AWS-account blueFill; classDef orange stroke-width:2px,color:#FFFFFF,stroke:#FF9D1C class adf1,adf2,cms1,cms2,AWS-account orange ``` ### CloudWatch roles and permissions To use CloudWatch Metric Streams in Observability Platform, you must configure a CloudWatch Metric Stream in each AWS account and region. The account you use to set up the CloudWatch Metric Stream must either have the `CloudWatchFullAccess` policy and `iam:PassRole` permission, or it must have the following list of permissions: * `iam:PassRole` * `cloudwatch:PutMetricStream` * `cloudwatch:DeleteMetricStream` * `cloudwatch:GetMetricStream` * `cloudwatch:ListMetricStreams` * `cloudwatch:StartMetricStreams` * `cloudwatch:StopMetricStreams` * `iam:CreateRole` * `iam:PutRolePolicy` ### Observability Platform authentication You must also create or use the API token of an Observability Platform restricted service account with write-only permission. For more information, see [Create a restricted service account](/administer/accounts-teams/service-accounts#create-a-restricted-service-account). You must also provide your Observability Platform organization name, which is the name of the subdomain that you use to access Observability Platform. For example, if your team uses `example.chronosphere.io`, your team's organization name is `example`. ### AWS resources and IAM roles The AWS setup process automatically creates the following resources and IAM roles as part of creating a metrics stream: #### AWS resources * **S3 Bucket:** A bucket will be created to store data processed by the Kinesis Firehose delivery stream. * **CloudWatch Log Group:** A log group will be created to capture logs related to the Kinesis Firehose delivery stream. * **Kinesis Firehose Delivery Stream:** A Kinesis Firehose delivery stream will be created with configurations to send data to Observability Platform through an HTTP endpoint, and store backup data in the S3 bucket. * **IAM Role for S3:** An IAM role will be created with the following permissions for the Kinesis Firehose to access the S3 bucket and CloudWatch Logs: * `s3:AbortMultipartUpload` * `s3:GetBucketLocation` * `s3:GetObject` * `s3:ListBucket` * `s3:ListBucketMultipartUploads` * `s3:PutObject` * `logs:PutLogEvents` #### IAM roles An IAM role to allow CloudWatch Metric Streams to publish data to the Kinesis Firehose delivery stream will be created with the following permissions: * `firehose:PutRecord` * `firehose:PutRecordBatch` ### Apply the configuration You can configure CloudWatch Metric Streams either manually in the AWS Management Console or by using Terraform. Before configuring metric ingestion, you can set up a drop rule to drop all metrics sent by CloudWatch Metric Streams. Doing this avoids unexpected license consumption changes. For examples, see [Drop CloudWatch Metric Stream metrics](#drop-cloudwatch-metric-stream-metrics). 1. Set values for the following environment variables, and modify the following Terraform data, and resources to apply the required settings. ```terraform expandable Terraform example icon="square-terminal" theme={null} # Variables variable "chronosphere_org_name" { type = string description = "The name of your Observability Platform organization, which is the subdomain name before .chronosphere.io." } variable "chronosphere_api_token" { type = string sensitive = true description = "The API token for an Observability Platform Restricted Service Account with write-only permission." } variable "failed_data_bucket_name" { type = string description = "The name of the S3 bucket to create to store data that couldn't be delivered to Observability Platform. If not specified, a random name will be generated." default = "" } variable "common_resource_attributes" { type = map(string) description = "Key-value pairs to apply as OpenTelemetry Resource Attributes on all metrics in this stream." default = {} } # IAM Policy Documents data "aws_iam_policy_document" "kinesis-firehose-stream-role-trust-policy" { statement { sid = "AllowRoleAssumptionByKinesisFirehose" effect = "Allow" actions = ["sts:AssumeRole"] principals { type = "Service" identifiers = ["firehose.amazonaws.com"] } } } data "aws_iam_policy_document" "kinesis-firehose-stream-role-s3-policy" { statement { sid = "AllowFirehoseS3Access" effect = "Allow" actions = [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUpload", "s3:PutObject" ] resources = [ aws_s3_bucket.kinesis-firehose-stream-failed-data.arn, "${aws_s3_bucket.kinesis-firehose-stream-failed-data.arn}/*" ] } } data "aws_iam_policy_document" "cloudwatch-metric-stream-role-trust-policy" { statement { sid = "AllowRoleAssumptionByloudWatchMetricStream" effect = "Allow" actions = ["sts:AssumeRole"] principals { type = "Service" identifiers = ["streams.metrics.cloudwatch.amazonaws.com"] } } } data "aws_iam_policy_document" "cloudwatch-metric-stream-role-firehose-policy" { statement { sid = "AllowCloudWatchFirehoseAccess" effect = "Allow" actions = [ "firehose:PutRecord", "firehose:PutRecordBatch" ] resources = [ aws_kinesis_firehose_delivery_stream.kinesis-firehose-stream.arn ] } } # S3 resource "random_id" "default_bucket_name_suffix" { byte_length = 8 } resource "aws_s3_bucket" "kinesis-firehose-stream-failed-data" { bucket = var.failed_data_bucket_name != "" ? var.failed_data_bucket_name : "chronosphere-cw-stream-failed-data-${random_id.default_bucket_name_suffix.hex}" } resource "aws_s3_bucket_public_access_block" "kinesis-firehose-stream-failed-data" { bucket = aws_s3_bucket.kinesis-firehose-stream-failed-data.id block_public_acls = true block_public_policy = true ignore_public_acls = true restrict_public_buckets = true } resource "aws_s3_bucket_server_side_encryption_configuration" "kinesis-firehose-stream-failed-data" { bucket = aws_s3_bucket.kinesis-firehose-stream-failed-data.id rule { apply_server_side_encryption_by_default { sse_algorithm = "AES256" } bucket_key_enabled = false } } resource "aws_s3_bucket_lifecycle_configuration" "kinesis-firehose-stream-failed-data" { bucket = aws_s3_bucket.kinesis-firehose-stream-failed-data.id rule { id = "Cleanup" expiration { days = 90 } status = "Enabled" } } # Kinesis Firehose Delivery Stream resource "aws_iam_role" "kinesis-firehose-stream-role" { name = "cloudwatch-firehose-stream-role" assume_role_policy = data.aws_iam_policy_document.kinesis-firehose-stream-role-trust-policy.json tags = { Name = "cloudwatch-firehose-stream-role" } } resource "aws_iam_role_policy" "kinesis-firehose-stream-role-s3-policy" { name = "KinesisFirehose-S3Access" role = aws_iam_role.kinesis-firehose-stream-role.id policy = data.aws_iam_policy_document.kinesis-firehose-stream-role-s3-policy.json } resource "aws_kinesis_firehose_delivery_stream" "kinesis-firehose-stream" { name = "chronosphere-cloudwatch-metric-stream" destination = "http_endpoint" http_endpoint_configuration { name = "chronosphere-http-endpoint" url = "https://${var.chronosphere_org_name}.chronosphere.io/data/metrics/api/v1/cloudwatch/firehose" access_key = var.chronosphere_api_token buffering_size = 1 # MiB buffering_interval = 60 # seconds role_arn = aws_iam_role.kinesis-firehose-stream-role.arn s3_backup_mode = "FailedDataOnly" retry_duration = 300 # seconds s3_configuration { role_arn = aws_iam_role.kinesis-firehose-stream-role.arn bucket_arn = aws_s3_bucket.kinesis-firehose-stream-failed-data.arn buffering_size = 10 # MiB buffering_interval = 300 # seconds compression_format = "GZIP" } request_configuration { content_encoding = "GZIP" dynamic "common_attributes" { for_each = var.common_resource_attributes iterator = attribute content { name = attribute.key value = attribute.value } } } } server_side_encryption { enabled = true } } # CloudWatch Metric Stream resource "aws_iam_role" "cloudwatch-metric-stream-role" { name = "cloudwatch-metric-stream-role" assume_role_policy = data.aws_iam_policy_document.cloudwatch-metric-stream-role-trust-policy.json tags = { Name = "cloudwatch-metric-stream-role" } } resource "aws_iam_role_policy" "cloudwatch-metric-stream-role-firehose-policy" { name = "MetricStreams-FirehosePutRecords" role = aws_iam_role.cloudwatch-metric-stream-role.id policy = data.aws_iam_policy_document.cloudwatch-metric-stream-role-firehose-policy.json } resource "aws_cloudwatch_metric_stream" "cloudwatch-metric-stream" { name = "chronosphere-metric-stream" role_arn = aws_iam_role.cloudwatch-metric-stream-role.arn firehose_arn = aws_kinesis_firehose_delivery_stream.kinesis-firehose-stream.arn output_format = "opentelemetry1.0" } ``` To configure CloudWatch Metric Streams using the AWS Management Console: 1. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/). 2. In the navigation pane, go to **Metrics > Streams**. 3. Click **Create metric stream**. 4. Click **Custom setup with Firehose**. 5. To create a new Amazon Data Firehose stream, click **Set up an Amazon Data Firehose stream** under **Select your Amazon Data Firehose stream**. 6. In **Create Firehose Stream**, set the following options: * **Set source:** `Amazon Kinesis Data Streams` or `Direct PUT` * **Destination:** `HTTP Endpoint` * **Firehose stream name:** `chronosphere-cloudwatch-metric-stream` * **Destination settings:** * **HTTP endpoint name:** `Chronosphere CloudWatch Endpoint` * **HTTP endpoint URL:** `https://.chronosphere.io/data/metrics/api/v1/cloudwatch/firehose`. Replace `` with your Observability Platform organization name. * **Authentication:** `Use access key` * **Access key:** Enter your Observability Platform Restricted Service Account's API token. * **Content encoding:** `GZIP` * **Retry duration:** `300 seconds` * Expand **Buffering Hints**. * **Buffer size:** `1 MiB` * **Buffer interval:** `60 seconds` * **Backup settings:** * **Source record backup in Amazon S3:** `Failed data only` * **S3 backup bucket:** Select a bucket, or create a new bucket. If you create a new bucket, provide a unique name and leave all default settings. * Expand **Buffer hints, compression and encryption**. * **Buffer size:** `10 MiB` * **Buffer interval:** `300 seconds` * **Compression for data records:** `GZIP` 7. Click **Create Firehose stream**. 8. Return to **Create a metric stream**. 9. In **Select your Amazon Data Firehose stream**, select `PUT-CW-STREAM-CHRONOSPHERE`. This is the Amazon Data Firehose stream you created previously. 10. Expand **Change output format** and verify that the output format is `OpenTelemetry 1.0`. 11. Under **Metrics to be streamed**, select **All metrics** to stream all CloudWatch metrics, or **Select metrics** to define which metrics to include or exclude in the metrics stream. 12. Set the **Custom metric stream name** to `chronosphere-cloudwatch-metric-stream`. 13. Click **Create metric stream**. ## Stream resource attributes CloudWatch Metric Streams include OpenTelemetry Protocol (OTLP) resource attributes, which Observability Platform merges into the time series. Observability Platform replaces periods (`.`) with underscores (`_`) in attribute key names. Amazon Data Firehose includes the following resource attributes in every post: * `aws_exporter_arn`: The Amazon Resource Name (ARN) of the CloudWatch Metric Stream, which serves as the unique metric writer instance identifier. * `cloud_account_id`: The account ID of the Amazon Data Firehose sending the stream, such as `123456789`. * `cloud_provider`: The value is always `aws`. * `cloud_region`: The AWS region of the Amazon Data Firehose sending the stream, such as `us-east-2`. For examples, see [Example metric names](#example-metric-names). ### Add custom resource attributes using stream parameters You can define custom key:value pairs as parameters for Amazon Data Firehose to include in each HTTP call. Observability Platform treats all additional parameters as resource attributes and merges them into the time series. Your custom parameters take precedence over the default CloudWatch metrics resource attributes. To avoid accidentally overwriting CloudWatch-provided values, do not add custom parameters with key names that conflict with default CloudWatch key names. You can configure additional parameters using either the AWS Management Console or Terraform. To configure parameters using Terraform: 1. Edit the Terraform module. 2. In the `aws_kinesis_firehose_delivery_stream` resource definition, modify the `request_configuration` block to define additional `common_attributes`. The AWS Management Console names this setting **Parameters**, while the API name is `common_attributes`. For example, this configures the `content_encoding` parameter to `GZIP` and defines two `common_attributes`, `testname` and `testname2`: ```terraform Terraform example icon="square-terminal" theme={null} request_configuration { content_encoding = "GZIP" common_attributes { name = "testname" value = "testvalue" } common_attributes { name = "testname2" value = "testvalue2" } } ``` To configure parameters using the AWS Management Console: 1. Navigate to the Amazon Data Firehose service. 2. Click the stream sending CloudWatch metrics (​​`PUT-CW-STREAM-CHRONOSPHERE`). 3. Click the **Configuration** tab in the Firehose stream details view. 4. In the Destination settings panel, click **Add parameters**. 5. In the Edit destination settings view, click **Add parameter**. 6. Enter the key and value. To add more parameters, click **Add parameter**. 7. Click **Save changes**. ### Example metric names Given a CloudWatch metric with the following attributes: * The namespace `AWS/EBS` (AWS Service) * The metric name `VolumeReadBytes` * The dimension `VolumeId` * The custom Firehose destination parameter `environment` * The CloudWatch metric resource attributes `aws_exporter_arn`, `cloud_account_id`, `cloud_provider`, and `cloud_region` Observability Platform creates metrics with these names and labels: ```text theme={null} aws_ebs_VolumeReadBytes_count{dimension_VolumeId="xyz", aws_exporter_arn="abc123", cloud_account_id="123", cloud_provider="aws", cloud_region="us-east-2", environment="staging"} aws_ebs_VolumeReadBytes_sum{dimension_VolumeId="xyz", aws_exporter_arn="abc123", cloud_account_id="123", cloud_provider="aws", cloud_region="us-east-2", environment="staging"} aws_ebs_VolumeReadBytes_maximum{dimension_VolumeId="xyz", aws_exporter_arn="abc123", cloud_account_id="123", cloud_provider="aws", cloud_region="us-east-2", environment="staging"} aws_ebs_VolumeReadBytes_minimum{dimension_VolumeId="xyz", aws_exporter_arn="abc123", cloud_account_id="123", cloud_provider="aws", cloud_region="us-east-2", environment="staging"} aws_ebs_VolumeReadBytes_average{dimension_VolumeId="xyz", aws_exporter_arn="abc123", cloud_account_id="123", cloud_provider="aws", cloud_region="us-east-2", environment="staging"} ``` Given a CloudWatch metric with the following attributes: * The namespace `Buildkite` (custom metrics) * The metric name `RunningJobsCount` * The custom Firehose destination parameter `environment` * No dimension * CloudWatch resource metric attributes `aws_exporter_arn`, `cloud_account_id`, `cloud_provider`, and `cloud_region` Observability Platform creates metrics with these names and labels: ```text theme={null} buildkite_RunningJobsCount_count{aws_exporter_arn="abc123", cloud_account_id="123", cloud_provider="aws", cloud_region="us-east-2", environment="staging"} buildkite_RunningJobsCount_sum{aws_exporter_arn="abc123", cloud_account_id="123", cloud_provider="aws", cloud_region="us-east-2", environment="staging"} buildkite_RunningJobsCount_maximum{aws_exporter_arn="abc123", cloud_account_id="123", cloud_provider="aws", cloud_region="us-east-2", environment="staging"} buildkite_RunningJobsCount_minimum{aws_exporter_arn="abc123", cloud_account_id="123", cloud_provider="aws", cloud_region="us-east-2", environment="staging"} buildkite_RunningJobsCount_average{aws_exporter_arn="abc123", cloud_account_id="123", cloud_provider="aws", cloud_region="us-east-2", environment="staging"} ``` ## Drop CloudWatch Metric Stream metrics When you ingest CloudWatch Metric Streams, you generate metrics that consume some of your Standard Metrics License capacity. To determine how this might affect license consumption, configure a [drop rule](/control/shaping/shape-metrics/rules/drop-rules) before configuring CloudWatch Metric Stream ingestion. ### Create rules to drop CloudWatch Metrics This example [Chronoctl](/tooling/chronoctl) YAML resource definition creates a drop rule that drops all metrics from CloudWatch Metric Streams *except* for metrics about the Metric Stream itself. ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: DropRule spec: slug: drop-cloudwatch-metric-stream-metrics name: Drop CloudWatch Metric Stream metrics mode: ENABLED filters: - name: aws_exporter_arn value_glob: arn:aws:cloudwatch:* - name: Namespace! value_glob: '{AWS/CloudWatch/MetricStreams}' ``` You can modify the rule to allow additional metrics from additional AWS namespaces. This example allows all metrics from the CloudWatch Metric Streams and AWS ECS namespaces. ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: DropRule spec: slug: drop-cloudwatch-metric-stream-metrics name: Drop CloudWatch Metric Stream metrics mode: ENABLED filters: - name: aws_exporter_arn value_glob: arn:aws:cloudwatch:* - name: Namespace! value_glob: '{AWS/CloudWatch/MetricStreams,AWS/ECS}' ``` You can also configure CloudWatch Metric Streams to include or exclude specific namespaces to reduce AWS costs associated with the streaming of unwanted metrics. 1. In the AWS Management Console, go to **Streams**. 2. Edit the Metric Stream. 3. Under **Metrics to be streamed**, include or exclude namespaces. ### View drop rule metrics To view how many data points per second that Observability Platform is dropping with the example CloudWatch Metric Streams drop rule, use the following PromQL query: ```text theme={null} sum by (policy_name) (rate(chrono_policies_count{dropped="yes", policy_name="Drop CloudWatch Metric Stream metrics"}[5m])) ``` ## Verify CloudWatch Metric Stream ingestion After setup, data can take from 5 to 10 minutes to arrive to Observability Platform. To verify functionality, check the operational dashboards in the AWS Management Console for the Metric Stream and Amazon Data Firehose. ### Check the CloudWatch Metrics Ingestion & Health dashboard The CloudWatch Metrics Ingestion & Health dashboard displays operational information about the health of your CloudWatch Metrics Streams integration with Observability Platform. The **CloudWatch Metric Streams** and **Data Firehose** panel groups rely on CloudWatch metrics sent from those services to Observability Platform. To populate these charts, include metrics from the `AWS/Firehose` and `AWS/CloudWatch/MetricStreams` namespaces in your CloudWatch Metrics Streams configuration. 1. In Observability Platform, go to **Dashboards**. 2. In the search bar, enter **CloudWatch**, and then click the **CloudWatch Metrics Ingestion & Health** dashboard. The **Observability Platform metrics ingestion** panel group displays information about the CloudWatch metrics Observability Platform received. 3. Check the **Data Firehose records received by Amazon Resource Name** chart to confirm that the ingestion API received Data Firehose records. 4. Check the **CloudWatch metric updates received** chart to confirm the number of CloudWatch metric updates Observability Platform extracts from the Data Firehose records. 5. Confirm that the **Transformed metrics** chart shows no **Rejected Data Points** 6. Check the **Unique time series by AWS metric namespace** chart to confirm that metrics from the AWS namespaces you want are in the stream that Observability Platform received. ### Check the Metric Stream Dashboard In the AWS Management Console, check the **Metric Updates** chart for specific metrics to validate that metrics are streaming. 1. In the AWS Management Console, go to **CloudWatch > Metric Streams**. 2. Click `chronosphere-cloudwatch-metric-stream` to view the status and operational statistics. 3. Verify that the **Status** is `Running`. 4. Verify how many updates have been sent in the **Metric Updates** chart. If the stream is working, the chart should report a non-zero number of updates. 5. Verify whether any errors were reported in the **Errors** chart. The value should be 0. ### Check the Amazon Data Firehose status In the AWS Management Console, check the status of several charts to ensure that metrics are streaming. 1. In the AWS Management Console, go to **Amazon Data Firehose > Firehose Streams**. 2. Select `PUT-CW-STREAM-CHRONOSPHERE` to view status and operational statistics for the Amazon Data Firehose. 3. The **Incoming bytes**, **Incoming put requests**, and **Incoming records charts** should all report non-zero values. 4. The **HTTP endpoint delivery success** chart should report a 100% successful metric count. 5. The **Records delivered to HTTP endpoint** chart should report a non-zero value. ### Query metrics about AWS CloudWatch Metric Streams You can also query for metrics about AWS CloudWatch Metric Streams in Observability Platform. These metrics won't appear in the Observability Platform Metrics Explorer if you've defined a drop rule to drop all AWS metrics. Modify the rule to allow some metrics, such as all metrics from the `AWS/CloudWatch/MetricStreams` namespace. For examples, see [Drop CloudWatch Metric Stream metrics](#drop-cloudwatch-metric-stream-metrics). 1. In Observability Platform, go to ** Explorers > Metrics Explorer**. 2. In the query box, enter `aws_cloudwatch_metricstreams` \`to view a list of AWS metrics received from the CloudWatch Metric Stream. 3. Select the metric you want to query to add it to the query prompt. You can then write a query around the selected metric. For example, run this query to report the rate of CloudWatch metric updates: ```text theme={null} rate(aws_cloudwatch_metricstreams_MetricUpdate_sum[5m]) ``` # Azure metrics integration Source: https://docs.chronosphere.io/ingest/metrics-traces/azure Learn how to ingest Azure metrics into Chronosphere Observability Platform. This feature isn't available to all Chronosphere Observability Platform users and might not be visible in your app. For information about enabling this feature in your environment, contact [Chronosphere Support](/support). [Azure metrics](https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/data-platform-metrics) integration connects Chronosphere Observability Platform with Azure Monitor to ingest metrics. ## Create the Azure principal Configure Azure to allow Observability Platform to access metrics in Azure. To do so: 1. Ensure the Azure Managed Identity principal is assigned the built-in [`Reader`](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/general#reader) role for every subscription within scope. The `Reader` role is more restrictive than the `MonitoringReader` role. It's important to create a dedicated Managed Identity to isolate the Azure API quota used by Chronosphere. 2. Contact [Chronosphere Support](/support) to obtain the specific `credential_issuer` and `credential_subject` for your tenant. 3. Copy both the `credential_issuer` and `credential_subject` variables exactly as specified to allow authentication between your Chronosphere tenant and Azure. 4. When creating the Azure principal, grant it access to subscriptions using one of the following options: * Management group-based: Access can be granted at the [Azure Management Group](https://learn.microsoft.com/en-us/azure/governance/management-groups/overview) level. This is the preferred approach, as it means that the subscriptions that Chronosphere has access to are kept in sync with the management group. As subscriptions are added and removed from the management group, these changes will be automatically reflected in Chronosphere. The same principal can also be granted access to multiple management groups. * Subscription-based: Access can be granted at the subscription level. In this case, the principal is granted access to individual subscriptions. This approach provides the most control, but has the downside that the list of subscriptions granted to the principal must be kept up to date by the customer. Chronosphere won't be able to ingest Azure metric data for subscriptions that haven't been assigned to the principal. 5. After provisioning the Azure Managed Identity, provide Chronosphere with the two Terraform outputs defined in the following examples: `azure_tenant_id` and `identity_client_id`. These outputs specify the ID of your Azure tenant and the client ID of the Azure Managed Identity, respectively. Chronosphere requires both values to complete the integration. ### Terraform examples The following examples show how you can integrate Azure using one of the previously mentioned methods: The following Terraform code provides an example of how to create an Azure principal and grant it access to the subscriptions within a management group. ```terraform expandable Terraform example icon="square-terminal" theme={null} ################################################################################ ## Azure Provider terraform { required_providers { azurerm = { source = "hashicorp/azurerm" version = "~> 4.0" } } } provider "azurerm" { subscription_id = var.subscription_id features {} } ################################################################################ ## Terraform Variables variable "subscription_id" { type = string description = "Azure subscription ID in which to create resources" } variable "location" { type = string description = "Location in which to create resources" default = "eastus" } variable "resource_group_name" { type = string description = "Name of the existing resource group to create resources in" } variable "management_group_id" { type = string description = "Management group to be integrated with Chronosphere" default = "production-group" } variable "principal_name" { type = string description = "Name of the Azure principal managed identity" default = "chronosphere-integration" } variable "credential_issuer" { type = string description = "Federated identity credential issuer" default = "https://accounts.google.com" } variable "credential_subject" { type = string description = "Unique ID of your tenant-specific GCP service account" default = "123456789123456789" } ################################################################################ ## Azure Resources data "azurerm_management_group" "this" { name = var.management_group_id } resource "azurerm_user_assigned_identity" "chronosphere" { name = var.principal_name location = var.location resource_group_name = var.resource_group_name } # Assign the principal the Reader role on all subscriptions in the management group. resource "azurerm_role_assignment" "chronosphere" { principal_id = azurerm_user_assigned_identity.chronosphere.principal_id scope = data.azurerm_management_group.this.id role_definition_name = "Reader" } resource "azurerm_federated_identity_credential" "chronosphere" { name = var.principal_name issuer = var.credential_issuer subject = var.credential_subject resource_group_name = var.resource_group_name parent_id = azurerm_user_assigned_identity.chronosphere.id audience = ["api://AzureADTokenExchange"] } ################################################################################ ## Terraform Outputs output "azure_tenant_id" { value = azurerm_user_assigned_identity.chronosphere.tenant_id } output "identity_client_id" { value = azurerm_user_assigned_identity.chronosphere.client_id } ``` The following Terraform code provides an example of how to create an Azure principal and grant it access to individual subscriptions. In this example, read access is granted to all accessible subscriptions. ```terraform expandable Terraform example icon="square-terminal" theme={null} ################################################################################ ## Azure Provider terraform { required_providers { azurerm = { source = "hashicorp/azurerm" version = "~> 4.0" } } } provider "azurerm" { subscription_id = var.subscription_id features {} } ################################################################################ ## Terraform Variables variable "subscription_id" { type = string description = "Azure subscription ID in which to create resources" } variable "location" { type = string description = "Location in which to create resources" default = "eastus" } variable "resource_group_name" { type = string description = "Name of the existing resource group to create resources in" } variable "principal_name" { type = string description = "Name of the Azure principal managed identity" default = "chronosphere-integration" } variable "credential_issuer" { type = string description = "Federated identity credential issuer" default = "https://accounts.google.com" } variable "credential_subject" { type = string description = "Unique ID of your tenant-specific GCP service account" default = "123456789123456789" } ################################################################################ ## Azure Resources data "azurerm_subscriptions" "all" {} resource "azurerm_user_assigned_identity" "chronosphere" { name = var.principal_name location = var.location resource_group_name = var.resource_group_name } # Assign the principal the Reader role on each subscription. resource "azurerm_role_assignment" "chronosphere" { for_each = toset(data.azurerm_subscriptions.all.subscriptions[*].id) principal_id = azurerm_user_assigned_identity.chronosphere.principal_id scope = each.key role_definition_name = "Reader" } resource "azurerm_federated_identity_credential" "chronosphere" { name = var.principal_name issuer = var.credential_issuer subject = var.credential_subject resource_group_name = var.resource_group_name parent_id = azurerm_user_assigned_identity.chronosphere.id audience = ["api://AzureADTokenExchange"] } ################################################################################ ## Terraform Outputs output "azure_tenant_id" { value = azurerm_user_assigned_identity.chronosphere.tenant_id } output "identity_client_id" { value = azurerm_user_assigned_identity.chronosphere.client_id } ``` ## Set up Observability Platform to receive Azure data After configuring Azure to enable access to metrics, you must configure Observability Platform to receive and process those metrics. To access the API directly, see the [Chronosphere API for Azure metrics](/tooling/api-info/definition/operations/ListAzureMetricsIntegrations). ### View Azure metrics integrations To list or view Azure metrics integrations, use one of the following options: To list your Azure metrics integrations using [Chronoctl](/tooling/chronoctl), use this command: ```shell theme={null} chronoctl azure-metrics-integrations list ``` To view a Azure metrics integration, use this command: ```shell theme={null} chronoctl azure-metrics-integrations read SLUG ``` Replace *`SLUG`* with the unique identifier of the Azure metrics integration. To list Azure metrics integrations with the Chronosphere API, use the [`ListAzureMetricsIntegrations`](/tooling/api-info/definition/operations/ListAzureMetricsIntegrations) endpoint. To view a single Azure metrics integration with the Chronosphere API, use the [`ReadAzureMetricsIntegration`](/tooling/api-info/definition/operations/ReadAzureMetricsIntegration) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Create or update an Azure metrics integration You can create or update your Azure metrics integration with Observability Platform by applying a configuration file with Chronoctl or Terraform. You must add your account principal to an Observability Platform team with [SysAdmin](/administer/accounts-teams/teams#add-a-role-to-a-team) permissions. To create a Azure metrics integration using [Chronoctl](/tooling/chronoctl), use this command: ```shell theme={null} chronoctl azure-metrics-integrations create --filename FILENAME ``` Replace *`FILENAME`* with the name of your Chronoctl configuration file. To update a Azure metrics integration, use this command: ```shell theme={null} chronoctl azure-metrics-integrations update --filename FILENAME ``` Replace *`FILENAME`* with the name of your Chronoctl configuration file. The input file uses the following structure: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: AzureMetricsIntegration spec: name: NAME slug: SLUG principal: tenant_id: TENANT_ID client_id: CLIENT_ID scrape_config: resource_types: - name: RESOURCE_TYPE_NAME metric_names: - METRIC_NAME locations: - LOCATION subscription_ids: - SUBSCRIPTION_ID usage_metrics_enabled: USAGE_METRICS_ENABLED count_metrics_enabled: COUNT_METRICS_ENABLED propagate_tags: PROPAGATE_TAGS ``` To create a Azure integration with Terraform, use the `chronosphere_azure_metrics_integration` resource: ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_azure_metrics_integration" "chronosphere_azure_metrics" { name = "Azure Metrics" slug = "azure-metrics" principal { tenant_id = TENANT_ID client_id = CLIENT_ID } scrape_config { resource_type { name = RESOURCE_TYPE_NAME metric_names = [METRIC_NAME] } locations = [LOCATION] subscription_ids = [SUBSCRIPTION_ID] } usage_metrics_enabled = USAGE_METRICS_ENABLED count_metrics_enabled = COUNT_METRICS_ENABLED propagate_tags = PROPAGATE_TAGS } ``` To create Azure integrations with the Observability Platform API, use the [`CreateAzureMetricsIntegration`](/tooling/api-info/definition/operations/CreateAzureMetricsIntegration) endpoint. To update a single Azure integration with the Observability Platform API, use the [`UpdateAzureMetricsIntegration`](/tooling/api-info/definition/operations/UpdateAzureMetricsIntegration) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. Replace the following: * *`NAME`*: (string) The name of the Azure integration. * *`SLUG`*: (string) The unique identifier of the Azure integration. * *`TENANT_ID`*: (UUID) The ID of the Azure tenant that hosts the managed identity principal. * *`CLIENT_ID`*: (UUID) The OAuth 2.0 client ID of the managed identity principal. * *`RESOURCE_TYPE_NAME`*: (string) Name of the resource type. * *`METRIC_NAME`*: (list(string)) List of metric names to be targeted. These apply to this resource type. Leave unset to retrieve all metrics. * *`LOCATION`*: (list(string)) Locations to be ingested for this integration. Applies to all subscriptions. Leave unset to retrieve all locations. * *`SUBSCRIPTION_ID`*: (list(string)) Subscriptions to be targeted for this integration. Leave unset to retrieve all subscriptions. * *`USAGE_METRICS_ENABLED`*: (Boolean) Enables collection of azure usage metrics under this principal (`Microsoft.Compute`, `Microsoft.Network`, `Microsoft.Storage`). * *`COUNT_METRICS_ENABLED`*: (Boolean) Enables Azure count metrics for the configured resources. * *`PROPAGATE_TAGS`*: (Boolean) Specifies whether Azure resource, group, and subscription tags should be propagated as metric labels. ### Delete an Azure integration Delete an Azure integration using one of the following methods: Your account must have [SysAdmin](/administer/accounts-teams/teams#add-a-role-to-a-team) permissions to complete this action. To delete an Azure integration using [Chronoctl](/tooling/chronoctl), use this command: Use the command: ```shell theme={null} chronoctl azure-metrics-integrations delete SLUG ``` Replace *`SLUG`* with the unique identifier of the Azure metrics integration. To complete this action with the Chronosphere API, use the [`DeleteAzureMetricsIntegration`](/tooling/api-info/definition/operations/DeleteAzureMetricsIntegration) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Metric information Observability Platform ingests a wide range of Azure metrics. See the [list of available Azure metrics](/ingest/metrics-traces/azure/azure-metrics). ### Metrics availability Azure metrics are displayed in Observability Platform with a several minute delay from their timestamps in Azure. On average, delays range anywhere from five minutes for most metrics to 10 minutes for very high cardinality metrics. These delays are caused by a number of factors external to Observability Platform, including each metric's scrape interval and latency within Azure. To accommodate these delays, use the PromQL [`offset` modifier](https://prometheus.io/docs/prometheus/latest/querying/basics/#offset-modifier) to [add offsets to your monitors](/investigate/alerts/troubleshooting#add-offsets-to-your-query). Additionally, delays can impact aggregation rules, and it might not be possible to aggregate metrics with particularly long delays. ### Metric labels You can request custom labels for your Azure metrics as `defaultLabels`. To add custom labels, contact [Chronosphere Support](/support). When importing metrics, some `defaultLabels` might conflict with prefixes which already exist in Observability Platform (for example, `job`). When this occurs, Observability Platform adds the prefix `exported_` to the source labels to prevent conflicts. ### Find Azure metrics in Metrics Explorer Use [Metrics Explorer](/investigate/querying/metrics/explorer) to find and review the status of your ingested metrics. * All Azure metrics start with the prefix `azure_`. Search for this prefix to display all Azure metrics in the platform. * Search supports substrings. For example, if the original Azure metric name contains a substring like `storageAccounts`, searching for the substring returns the Azure metric, along with other metrics containing the substring. # Azure metrics available in Chronosphere Observability Platform Source: https://docs.chronosphere.io/ingest/metrics-traces/azure/azure-metrics Reference the Azure metrics that Chronosphere Observability Platform ingests. Observability Platform ingests the following metrics for Azure. If there are additional metrics you want to ingest, contact [Chronosphere Support](/support). By default, all metrics are scraped. Users can customize the allowed metrics by customizing [the scrape configuration](/tooling/api-info/definition/operations/CreateAzureMetricsIntegration). For specific information about each metric, see Microsoft's documentation for [Supported metrics with AzureMonitor](https://learn.microsoft.com/en-us/azure/azure-monitor/reference/metrics-index). ## microsoft.analysisservices/servers * `CleanerCurrentPrice` * `CleanerMemoryNonshrinkable` * `CleanerMemoryShrinkable` * `CommandPoolBusyThreads` * `CommandPoolIdleThreads` * `CommandPoolJobQueueLength` * `CurrentConnections` * `CurrentUserSessions` * `LongParsingBusyThreads` * `LongParsingIdleThreads` * `LongParsingJobQueueLength` * `MemoryLimitHard` * `MemoryLimitHigh` * `MemoryLimitLow` * `MemoryLimitVertiPaq` * `MemoryUsage` * `ProcessingPoolBusyIOJobThreads` * `ProcessingPoolBusyNonIOThreads` * `ProcessingPoolIOJobQueueLength` * `ProcessingPoolIdleIOJobThreads` * `ProcessingPoolIdleNonIOThreads` * `ProcessingPoolJobQueueLength` * `QueryPoolBusyThreads` * `QueryPoolIdleThreads` * `QueryPoolJobQueueLength` * `Quota` * `QuotaBlocked` * `RowsConvertedPerSec` * `RowsReadPerSec` * `RowsWrittenPerSec` * `ShortParsingBusyThreads` * `ShortParsingIdleThreads` * `ShortParsingJobQueueLength` * `SuccessfullConnectionsPerSec` * `TotalConnectionFailures` * `TotalConnectionRequests` * `VertiPaqNonpaged` * `VertiPaqPaged` * `mashup_engine_memory_metric` * `mashup_engine_private_bytes_metric` * `mashup_engine_qpu_metric` * `mashup_engine_virtual_bytes_metric` * `memory_metric` * `memory_thrashing_metric` * `private_bytes_metric` * `qpu_metric` * `virtual_bytes_metric` ## microsoft.apimanagement/service * `BackendDuration` * `Capacity` * `ConnectionAttempts` * `CpuPercent_Gateway` * `Duration` * `EventHubDroppedEvents` * `EventHubRejectedEvents` * `EventHubSuccessfulEvents` * `EventHubThrottledEvents` * `EventHubTimedoutEvents` * `EventHubTotalBytesSent` * `EventHubTotalEvents` * `EventHubTotalFailedEvents` * `FailedRequests` * `MemoryPercent_Gateway` * `NetworkConnectivity` * `OtherRequests` * `Requests` * `SuccessfulRequests` * `TotalRequests` * `UnauthorizedRequests` * `WebSocketMessages` ## microsoft.app/containerapps * `CoresQuotaUsed` * `CpuPercentage` * `GpuUtilizationPercentage` * `JvmBufferCount` * `JvmBufferMemoryLimit` * `JvmBufferMemoryUsage` * `JvmGcCount` * `JvmGcDuration` * `JvmMemoryCommitted` * `JvmMemoryLimit` * `JvmMemoryTotalCommitted` * `JvmMemoryTotalLimit` * `JvmMemoryTotalUsed` * `JvmMemoryUsed` * `JvmThreadCount` * `MemoryPercentage` * `Replicas` * `Requests` * `ResiliencyConnectTimeouts` * `ResiliencyEjectedHosts` * `ResiliencyEjectionsAborted` * `ResiliencyRequestRetries` * `ResiliencyRequestTimeouts` * `ResiliencyRequestsPendingConnectionPool` * `ResponseTime` * `RestartCount` * `RxBytes` * `TotalCoresQuotaUsed` * `TxBytes` * `UsageNanoCores` * `WorkingSetBytes` ## microsoft.appconfiguration/configurationstores * `DailyStorageUsage` * `HttpIncomingRequestCount` * `HttpIncomingRequestDuration` * `ReplicationLatency` * `RequestQuotaUsage` * `SnapshotStorageSize` * `ThrottledHttpRequestCount` ## microsoft.automation/automationaccounts * `HybridWorkerPing` * `TotalJob` * `TotalUpdateDeploymentMachineRuns` * `TotalUpdateDeploymentRuns` ## microsoft.batch/batchaccounts * `CoreCount` * `CreatingNodeCount` * `IdleNodeCount` * `JobDeleteCompleteEvent` * `JobDeleteStartEvent` * `JobDisableCompleteEvent` * `JobDisableStartEvent` * `JobStartEvent` * `JobTerminateCompleteEvent` * `JobTerminateStartEvent` * `LeavingPoolNodeCount` * `LowPriorityCoreCount` * `OfflineNodeCount` * `PoolCreateEvent` * `PoolDeleteCompleteEvent` * `PoolDeleteStartEvent` * `PoolResizeCompleteEvent` * `PoolResizeStartEvent` * `PreemptedNodeCount` * `RebootingNodeCount` * `ReimagingNodeCount` * `RunningNodeCount` * `StartTaskFailedNodeCount` * `StartingNodeCount` * `TaskCompleteEvent` * `TaskFailEvent` * `TaskStartEvent` * `TotalLowPriorityNodeCount` * `TotalNodeCount` * `UnusableNodeCount` * `WaitingForStartTaskNodeCount` ## microsoft.cache/redis * `ConnectedClientsUsingAADToken` * `GeoReplicationConnectivityLag` * `GeoReplicationDataSyncOffset` * `GeoReplicationFullSyncEventFinished` * `GeoReplicationFullSyncEventStarted` * `GeoReplicationHealthy` * `LatencyP99` * `allConnectionsClosedPerSecond` * `allConnectionsCreatedPerSecond` * `allcacheRead` * `allcacheWrite` * `allcachehits` * `allcachemisses` * `allconnectedclients` * `allevictedkeys` * `allexpiredkeys` * `allgetcommands` * `alloperationsPerSecond` * `allpercentprocessortime` * `allserverLoad` * `allsetcommands` * `alltotalcommandsprocessed` * `alltotalkeys` * `allusedmemory` * `allusedmemoryRss` * `allusedmemorypercentage` * `cacheLatency` * `cacheRead` * `cacheRead0` * `cacheRead1` * `cacheRead2` * `cacheRead3` * `cacheRead4` * `cacheRead5` * `cacheRead6` * `cacheRead7` * `cacheRead8` * `cacheRead9` * `cacheWrite` * `cacheWrite0` * `cacheWrite1` * `cacheWrite2` * `cacheWrite3` * `cacheWrite4` * `cacheWrite5` * `cacheWrite6` * `cacheWrite7` * `cacheWrite8` * `cacheWrite9` * `cachehits` * `cachehits0` * `cachehits1` * `cachehits2` * `cachehits3` * `cachehits4` * `cachehits5` * `cachehits6` * `cachehits7` * `cachehits8` * `cachehits9` * `cachemisses` * `cachemisses0` * `cachemisses1` * `cachemisses2` * `cachemisses3` * `cachemisses4` * `cachemisses5` * `cachemisses6` * `cachemisses7` * `cachemisses8` * `cachemisses9` * `cachemissrate` * `connectedclients` * `connectedclients0` * `connectedclients1` * `connectedclients2` * `connectedclients3` * `connectedclients4` * `connectedclients5` * `connectedclients6` * `connectedclients7` * `connectedclients8` * `connectedclients9` * `errors` * `evictedkeys` * `evictedkeys0` * `evictedkeys1` * `evictedkeys2` * `evictedkeys3` * `evictedkeys4` * `evictedkeys5` * `evictedkeys6` * `evictedkeys7` * `evictedkeys8` * `evictedkeys9` * `expiredkeys` * `expiredkeys0` * `expiredkeys1` * `expiredkeys2` * `expiredkeys3` * `expiredkeys4` * `expiredkeys5` * `expiredkeys6` * `expiredkeys7` * `expiredkeys8` * `expiredkeys9` * `getcommands` * `getcommands0` * `getcommands1` * `getcommands2` * `getcommands3` * `getcommands4` * `getcommands5` * `getcommands6` * `getcommands7` * `getcommands8` * `getcommands9` * `operationsPerSecond` * `operationsPerSecond0` * `operationsPerSecond1` * `operationsPerSecond2` * `operationsPerSecond3` * `operationsPerSecond4` * `operationsPerSecond5` * `operationsPerSecond6` * `operationsPerSecond7` * `operationsPerSecond8` * `operationsPerSecond9` * `percentProcessorTime` * `percentProcessorTime0` * `percentProcessorTime1` * `percentProcessorTime2` * `percentProcessorTime3` * `percentProcessorTime4` * `percentProcessorTime5` * `percentProcessorTime6` * `percentProcessorTime7` * `percentProcessorTime8` * `percentProcessorTime9` * `serverLoad` * `serverLoad0` * `serverLoad1` * `serverLoad2` * `serverLoad3` * `serverLoad4` * `serverLoad5` * `serverLoad6` * `serverLoad7` * `serverLoad8` * `serverLoad9` * `setcommands` * `setcommands0` * `setcommands1` * `setcommands2` * `setcommands3` * `setcommands4` * `setcommands5` * `setcommands6` * `setcommands7` * `setcommands8` * `setcommands9` * `totalcommandsprocessed` * `totalcommandsprocessed0` * `totalcommandsprocessed1` * `totalcommandsprocessed2` * `totalcommandsprocessed3` * `totalcommandsprocessed4` * `totalcommandsprocessed5` * `totalcommandsprocessed6` * `totalcommandsprocessed7` * `totalcommandsprocessed8` * `totalcommandsprocessed9` * `totalkeys` * `totalkeys0` * `totalkeys1` * `totalkeys2` * `totalkeys3` * `totalkeys4` * `totalkeys5` * `totalkeys6` * `totalkeys7` * `totalkeys8` * `totalkeys9` * `usedmemory` * `usedmemory0` * `usedmemory1` * `usedmemory2` * `usedmemory3` * `usedmemory4` * `usedmemory5` * `usedmemory6` * `usedmemory7` * `usedmemory8` * `usedmemory9` * `usedmemoryRss` * `usedmemoryRss0` * `usedmemoryRss1` * `usedmemoryRss2` * `usedmemoryRss3` * `usedmemoryRss4` * `usedmemoryRss5` * `usedmemoryRss6` * `usedmemoryRss7` * `usedmemoryRss8` * `usedmemoryRss9` * `usedmemorypercentage` ## microsoft.cache/redisenterprise * `cacheLatency` * `cacheRead` * `cacheWrite` * `cachehits` * `cachemisses` * `connectedclients` * `evictedkeys` * `expiredkeys` * `geoReplicationHealthy` * `getcommands` * `operationsPerSecond` * `percentProcessorTime` * `serverLoad` * `setcommands` * `totalcommandsprocessed` * `totalkeys` * `usedmemory` * `usedmemorypercentage` ## microsoft.cdn/profiles * `ActiveWebSocketConnections` * `AverageWebSocketConnectionDuration` * `ByteHitRatio` * `OriginHealthPercentage` * `OriginLatency` * `OriginRequestCount` * `OriginShieldOriginRequestCount` * `OriginShieldRateLimitRequestCount` * `OriginShieldRequestCount` * `OriginShieldRequestSize` * `Percentage4XX` * `Percentage5XX` * `RequestCount` * `RequestSize` * `ResponseSize` * `TotalLatency` * `WebApplicationFirewallCaptchaRequestCount` * `WebApplicationFirewallJsRequestCount` * `WebApplicationFirewallRequestCount` * `WebSocketConnections` ## microsoft.cognitiveservices/accounts * `ActionFeatureIdOccurrences` * `ActionFeaturesPerEvent` * `ActionIdOccurrences` * `ActionNamespacesPerEvent` * `ActionsPerEvent` * `ActiveTokens` * `AnnotatedPages` * `AudioCompletionTokens` * `AudioInputTokens` * `AudioOutputTokens` * `AudioPromptTokens` * `AudioSecondsBatchTranscribed` * `AudioSecondsBatchWhisperTranscribed` * `AudioSecondsFastTranscribed` * `AudioSecondsFastWhisperTranscribed` * `AudioSecondsTranscribed` * `AudioSecondsTranslated` * `AvatarModelHostingSeconds` * `AvatarModelTrainingSeconds` * `AzureOpenAIAvailabilityRate` * `AzureOpenAIContextTokensCacheMatchRate` * `AzureOpenAINormalizedTBTInMS` * `AzureOpenAINormalizedTTFTInMS` * `AzureOpenAIProvisionedManagedUtilization` * `AzureOpenAIProvisionedManagedUtilizationV2` * `AzureOpenAIRequests` * `AzureOpenAITTLTInMS` * `AzureOpenAITimeToResponse` * `AzureOpenAITokenPerSecond` * `BaselineEstimatorOverallReward` * `BaselineEstimatorSlotReward` * `BaselineRandomEstimatorOverallReward` * `BaselineRandomEstimatorSlotReward` * `BaselineRandomEventCount` * `BaselineRandomReward` * `BlockedCalls` * `CarnegieInferenceCount` * `CharactersTrained` * `CharactersTranslated` * `ClientErrors` * `ComputerVisionTransactions` * `ContentSafetyImageAnalyzeRequestCount` * `ContentSafetyTextAnalyzeRequestCount` * `ContextFeatureIdOccurrences` * `ContextFeaturesPerEvent` * `ContextNamespacesPerEvent` * `CustomVisionTrainingTime` * `CustomVisionTransactions` * `DataIn` * `DataOut` * `DocumentCharactersTranslated` * `DocumentCustomCharactersTranslated` * `FaceImagesTrained` * `FaceTransactions` * `FacesStored` * `FeatureCardinality_Action` * `FeatureCardinality_Context` * `FeatureCardinality_Slot` * `FineTunedTrainingHours` * `GeneratedImages` * `GeneratedTokens` * `ImagesStored` * `InputTokens` * `JobDuration` * `LUISSpeechRequests` * `LUISTextRequests` * `Latency` * `LearnedEvents` * `MatchedRewards` * `ModelAvailabilityRate` * `ModelRequests` * `NonActivatedEvents` * `NormalizedTimeBetweenTokens` * `NormalizedTimeToFirstToken` * `NumberOfSlots` * `NumberofSpeakerProfiles` * `ObservedRewards` * `OneDocumentCharactersTranslated` * `OneDocumentCustomCharactersTranslated` * `OnlineEstimatorOverallReward` * `OnlineEstimatorSlotReward` * `OnlineEventCount` * `OnlineReward` * `OutputTokens` * `ProcessedCharacters` * `ProcessedHealthTextRecords` * `ProcessedImages` * `ProcessedPages` * `ProcessedPromptTokens` * `ProcessedTextRecords` * `ProvisionedUtilization` * `QuestionAnsweringTextRecords` * `RAIAbusiveUsersCount` * `RAIHarmfulRequests` * `RAIRejectedRequests` * `RAISystemEvent` * `RAITotalRequests` * `Ratelimit` * `RealtimeUsageTime` * `Reward` * `ServerErrors` * `SlotFeatureIdOccurrences` * `SlotFeaturesPerEvent` * `SlotIdOccurrences` * `SlotNamespacesPerEvent` * `SlotReward` * `SpeakerRecognitionTransactions` * `SpeechModelHostingHours` * `SpeechSessionDuration` * `SuccessRate` * `SuccessfulCalls` * `SynthesizedCharacters` * `TextCharactersTranslated` * `TextCustomCharactersTranslated` * `TextTrainedCharacters` * `TimeToLastByte` * `TimeToResponse` * `TokenTransaction` * `TokensPerSecond` * `TotalCalls` * `TotalErrors` * `TotalEvents` * `TotalPages` * `TotalTokenCalls` * `TotalTokens` * `TotalTransactions` * `TranslatorProAppSeconds` * `UserBaselineEventCount` * `UserBaselineReward` * `VideoSecondsSynthesized` * `VoiceLiveAudioInputTokens` * `VoiceLiveAudioOutputTokens` * `VoiceLiveCachedAudioInputTokens` * `VoiceLiveCachedTextInputTokens` * `VoiceLiveTextInputTokens` * `VoiceLiveTextOutputTokens` * `VoiceModelHostingHours` * `VoiceModelTrainingMinutes` * `cacheReadInputTokens` * `ephemeral1hInputTokens` * `ephemeral5mInputTokens` ## microsoft.compute/virtualmachines * `Available Memory Bytes` * `Available Memory Percentage` * `CPU Credits Consumed` * `CPU Credits Remaining` * `Data Disk Bandwidth Consumed Percentage` * `Data Disk IOPS Consumed Percentage` * `Data Disk Latency` * `Data Disk Max Burst Bandwidth` * `Data Disk Max Burst IOPS` * `Data Disk Queue Depth` * `Data Disk Read Bytes/sec` * `Data Disk Read Operations/Sec` * `Data Disk Target Bandwidth` * `Data Disk Target IOPS` * `Data Disk Used Burst BPS Credits Percentage` * `Data Disk Used Burst IO Credits Percentage` * `Data Disk Write Bytes/sec` * `Data Disk Write Operations/Sec` * `Disk Read Bytes` * `Disk Read Operations/Sec` * `Disk Write Bytes` * `Disk Write Operations/Sec` * `Inbound Flows` * `Inbound Flows Maximum Creation Rate` * `Network In` * `Network In Total` * `Network Out` * `Network Out Total` * `OS Disk Bandwidth Consumed Percentage` * `OS Disk IOPS Consumed Percentage` * `OS Disk Latency` * `OS Disk Max Burst Bandwidth` * `OS Disk Max Burst IOPS` * `OS Disk Queue Depth` * `OS Disk Read Bytes/sec` * `OS Disk Read Operations/Sec` * `OS Disk Target Bandwidth` * `OS Disk Target IOPS` * `OS Disk Used Burst BPS Credits Percentage` * `OS Disk Used Burst IO Credits Percentage` * `OS Disk Write Bytes/sec` * `OS Disk Write Operations/Sec` * `Outbound Flows` * `Outbound Flows Maximum Creation Rate` * `Percentage CPU` * `Premium Data Disk Cache Read Hit` * `Premium Data Disk Cache Read Miss` * `Premium OS Disk Cache Read Hit` * `Premium OS Disk Cache Read Miss` * `Temp Disk Latency` * `Temp Disk Queue Depth` * `Temp Disk Read Bytes/sec` * `Temp Disk Read Operations/Sec` * `Temp Disk Write Bytes/sec` * `Temp Disk Write Operations/Sec` * `VM Cached Bandwidth Consumed Percentage` * `VM Cached IOPS Consumed Percentage` * `VM Local Used Burst BPS Credits Percentage` * `VM Local Used Burst IO Credits Percentage` * `VM Remote Used Burst BPS Credits Percentage` * `VM Remote Used Burst IO Credits Percentage` * `VM Uncached Bandwidth Consumed Percentage` * `VM Uncached IOPS Consumed Percentage` * `VmAvailabilityMetric` ## microsoft.compute/virtualmachinescalesets * `Available Memory Bytes` * `Available Memory Percentage` * `CPU Credits Consumed` * `CPU Credits Remaining` * `Data Disk Bandwidth Consumed Percentage` * `Data Disk IOPS Consumed Percentage` * `Data Disk Latency` * `Data Disk Max Burst Bandwidth` * `Data Disk Max Burst IOPS` * `Data Disk Queue Depth` * `Data Disk Read Bytes/sec` * `Data Disk Read Operations/Sec` * `Data Disk Target Bandwidth` * `Data Disk Target IOPS` * `Data Disk Used Burst BPS Credits Percentage` * `Data Disk Used Burst IO Credits Percentage` * `Data Disk Write Bytes/sec` * `Data Disk Write Operations/Sec` * `Disk Read Bytes` * `Disk Read Operations/Sec` * `Disk Write Bytes` * `Disk Write Operations/Sec` * `Inbound Flows` * `Inbound Flows Maximum Creation Rate` * `Network In` * `Network In Total` * `Network Out` * `Network Out Total` * `OS Disk Bandwidth Consumed Percentage` * `OS Disk IOPS Consumed Percentage` * `OS Disk Latency` * `OS Disk Max Burst Bandwidth` * `OS Disk Max Burst IOPS` * `OS Disk Queue Depth` * `OS Disk Read Bytes/sec` * `OS Disk Read Operations/Sec` * `OS Disk Target Bandwidth` * `OS Disk Target IOPS` * `OS Disk Used Burst BPS Credits Percentage` * `OS Disk Used Burst IO Credits Percentage` * `OS Disk Write Bytes/sec` * `OS Disk Write Operations/Sec` * `Outbound Flows` * `Outbound Flows Maximum Creation Rate` * `Percentage CPU` * `Premium Data Disk Cache Read Hit` * `Premium Data Disk Cache Read Miss` * `Premium OS Disk Cache Read Hit` * `Premium OS Disk Cache Read Miss` * `Temp Disk Latency` * `Temp Disk Queue Depth` * `Temp Disk Read Bytes/sec` * `Temp Disk Read Operations/Sec` * `Temp Disk Write Bytes/sec` * `Temp Disk Write Operations/Sec` * `VM Cached Bandwidth Consumed Percentage` * `VM Cached IOPS Consumed Percentage` * `VM Local Used Burst BPS Credits Percentage` * `VM Local Used Burst IO Credits Percentage` * `VM Remote Used Burst BPS Credits Percentage` * `VM Remote Used Burst IO Credits Percentage` * `VM Uncached Bandwidth Consumed Percentage` * `VM Uncached IOPS Consumed Percentage` * `VmAvailabilityMetric` ## microsoft.compute/virtualmachinescalesets/virtualmachines * `Available Memory Bytes` * `Available Memory Percentage` * `CPU Credits Consumed` * `CPU Credits Remaining` * `Data Disk Bandwidth Consumed Percentage` * `Data Disk IOPS Consumed Percentage` * `Data Disk Latency` * `Data Disk Max Burst Bandwidth` * `Data Disk Max Burst IOPS` * `Data Disk Queue Depth` * `Data Disk Read Bytes/sec` * `Data Disk Read Operations/Sec` * `Data Disk Target Bandwidth` * `Data Disk Target IOPS` * `Data Disk Used Burst BPS Credits Percentage` * `Data Disk Used Burst IO Credits Percentage` * `Data Disk Write Bytes/sec` * `Data Disk Write Operations/Sec` * `Disk Read Bytes` * `Disk Read Operations/Sec` * `Disk Write Bytes` * `Disk Write Operations/Sec` * `Inbound Flows` * `Inbound Flows Maximum Creation Rate` * `Network In` * `Network In Total` * `Network Out` * `Network Out Total` * `OS Disk Bandwidth Consumed Percentage` * `OS Disk IOPS Consumed Percentage` * `OS Disk Latency` * `OS Disk Max Burst Bandwidth` * `OS Disk Max Burst IOPS` * `OS Disk Queue Depth` * `OS Disk Read Bytes/sec` * `OS Disk Read Operations/Sec` * `OS Disk Target Bandwidth` * `OS Disk Target IOPS` * `OS Disk Used Burst BPS Credits Percentage` * `OS Disk Used Burst IO Credits Percentage` * `OS Disk Write Bytes/sec` * `OS Disk Write Operations/Sec` * `Outbound Flows` * `Outbound Flows Maximum Creation Rate` * `Percentage CPU` * `Premium Data Disk Cache Read Hit` * `Premium Data Disk Cache Read Miss` * `Premium OS Disk Cache Read Hit` * `Premium OS Disk Cache Read Miss` * `Temp Disk Latency` * `Temp Disk Queue Depth` * `Temp Disk Read Bytes/sec` * `Temp Disk Read Operations/Sec` * `Temp Disk Write Bytes/sec` * `Temp Disk Write Operations/Sec` * `VM Cached Bandwidth Consumed Percentage` * `VM Cached IOPS Consumed Percentage` * `VM Local Used Burst BPS Credits Percentage` * `VM Local Used Burst IO Credits Percentage` * `VM Remote Used Burst BPS Credits Percentage` * `VM Remote Used Burst IO Credits Percentage` * `VM Uncached Bandwidth Consumed Percentage` * `VM Uncached IOPS Consumed Percentage` * `VmAvailabilityMetric` ## microsoft.containerinstance/containergroups * `CpuUsage` * `MemoryUsage` * `NetworkBytesReceivedPerSecond` * `NetworkBytesTransmittedPerSecond` ## microsoft.containerregistry/registries * `AgentPoolCPUTime` * `RunDuration` * `StorageUsed` * `SuccessfulPullCount` * `SuccessfulPushCount` * `TotalPullCount` * `TotalPushCount` ## microsoft.containerservice/managedclusters * `apiserver_cpu_usage_percentage` * `apiserver_current_inflight_requests` * `apiserver_memory_usage_percentage` * `cluster_autoscaler_cluster_safe_to_autoscale` * `cluster_autoscaler_scale_down_in_cooldown` * `cluster_autoscaler_unneeded_nodes_count` * `cluster_autoscaler_unschedulable_pods_count` * `etcd_cpu_usage_percentage` * `etcd_database_usage_percentage` * `etcd_memory_usage_percentage` * `kube_node_status_allocatable_cpu_cores` * `kube_node_status_allocatable_memory_bytes` * `kube_node_status_condition` * `node_cpu_usage_millicores` * `node_cpu_usage_percentage` * `node_disk_usage_bytes` * `node_disk_usage_percentage` * `node_memory_rss_bytes` * `node_memory_rss_percentage` * `node_memory_working_set_bytes` * `node_memory_working_set_percentage` * `node_network_in_bytes` * `node_network_out_bytes` ## microsoft.datafactory/factories * `ActivityCancelledRuns` * `ActivityFailedRuns` * `ActivitySucceededRuns` * `AirflowIntegrationRuntimeCeleryTaskTimeoutError` * `AirflowIntegrationRuntimeCollectDBDags` * `AirflowIntegrationRuntimeCpuPercentage` * `AirflowIntegrationRuntimeCpuUsage` * `AirflowIntegrationRuntimeDAGFileRefreshError` * `AirflowIntegrationRuntimeDAGProcessingImportErrors` * `AirflowIntegrationRuntimeDAGProcessingLastDuration` * `AirflowIntegrationRuntimeDAGProcessingLastRunSecondsAgo` * `AirflowIntegrationRuntimeDAGProcessingManagerStalls` * `AirflowIntegrationRuntimeDAGProcessingProcesses` * `AirflowIntegrationRuntimeDAGProcessingProcessorTimeouts` * `AirflowIntegrationRuntimeDAGProcessingTotalParseTime` * `AirflowIntegrationRuntimeDAGRunDependencyCheck` * `AirflowIntegrationRuntimeDAGRunDurationFailed` * `AirflowIntegrationRuntimeDAGRunDurationSuccess` * `AirflowIntegrationRuntimeDAGRunFirstTaskSchedulingDelay` * `AirflowIntegrationRuntimeDAGRunScheduleDelay` * `AirflowIntegrationRuntimeDagBagSize` * `AirflowIntegrationRuntimeDagCallbackExceptions` * `AirflowIntegrationRuntimeExecutorOpenSlots` * `AirflowIntegrationRuntimeExecutorQueuedTasks` * `AirflowIntegrationRuntimeExecutorRunningTasks` * `AirflowIntegrationRuntimeJobEnd` * `AirflowIntegrationRuntimeJobHeartbeatFailure` * `AirflowIntegrationRuntimeJobStart` * `AirflowIntegrationRuntimeMemoryPercentage` * `AirflowIntegrationRuntimeMemoryUsage` * `AirflowIntegrationRuntimeNodeCount` * `AirflowIntegrationRuntimeOperatorFailures` * `AirflowIntegrationRuntimeOperatorSuccesses` * `AirflowIntegrationRuntimePoolOpenSlots` * `AirflowIntegrationRuntimePoolQueuedSlots` * `AirflowIntegrationRuntimePoolRunningSlots` * `AirflowIntegrationRuntimePoolStarvingTasks` * `AirflowIntegrationRuntimeSchedulerCriticalSectionBusy` * `AirflowIntegrationRuntimeSchedulerCriticalSectionDuration` * `AirflowIntegrationRuntimeSchedulerFailedSLAEmailAttempts` * `AirflowIntegrationRuntimeSchedulerHeartbeat` * `AirflowIntegrationRuntimeSchedulerOrphanedTasksAdopted` * `AirflowIntegrationRuntimeSchedulerOrphanedTasksCleared` * `AirflowIntegrationRuntimeSchedulerTasksExecutable` * `AirflowIntegrationRuntimeSchedulerTasksKilledExternally` * `AirflowIntegrationRuntimeSchedulerTasksRunning` * `AirflowIntegrationRuntimeSchedulerTasksStarving` * `AirflowIntegrationRuntimeStartedTaskInstances` * `AirflowIntegrationRuntimeTaskInstanceCreatedUsingOperator` * `AirflowIntegrationRuntimeTaskInstanceDuration` * `AirflowIntegrationRuntimeTaskInstanceFailures` * `AirflowIntegrationRuntimeTaskInstanceFinished` * `AirflowIntegrationRuntimeTaskInstancePreviouslySucceeded` * `AirflowIntegrationRuntimeTaskInstanceSuccesses` * `AirflowIntegrationRuntimeTaskRemovedFromDAG` * `AirflowIntegrationRuntimeTaskRestoredToDAG` * `AirflowIntegrationRuntimeTriggersBlockedMainThread` * `AirflowIntegrationRuntimeTriggersFailed` * `AirflowIntegrationRuntimeTriggersRunning` * `AirflowIntegrationRuntimeTriggersSucceeded` * `AirflowIntegrationRuntimeZombiesKilled` * `FactorySizeInGbUnits` * `IntegrationRuntimeAvailableMemory` * `IntegrationRuntimeAvailableNodeNumber` * `IntegrationRuntimeAverageTaskPickupDelay` * `IntegrationRuntimeCpuPercentage` * `IntegrationRuntimeQueueLength` * `MVNetIRCopyAvailableCapacityPCT` * `MVNetIRCopyCapacityUtilization` * `MVNetIRCopyWaitingQueueLength` * `MVNetIRExternalAvailableCapacityPCT` * `MVNetIRExternalCapacityUtilization` * `MVNetIRExternalWaitingQueueLength` * `MVNetIRPipelineAvailableCapacityPCT` * `MVNetIRPipelineCapacityUtilization` * `MVNetIRPipelineWaitingQueueLength` * `MaxAllowedFactorySizeInGbUnits` * `MaxAllowedResourceCount` * `PipelineCancelledRuns` * `PipelineElapsedTimeRuns` * `PipelineFailedRuns` * `PipelineSucceededRuns` * `ResourceCount` * `SSISIntegrationRuntimeStartCancel` * `SSISIntegrationRuntimeStartFailed` * `SSISIntegrationRuntimeStartSucceeded` * `SSISIntegrationRuntimeStopStuck` * `SSISIntegrationRuntimeStopSucceeded` * `SSISPackageExecutionCancel` * `SSISPackageExecutionFailed` * `SSISPackageExecutionSucceeded` * `TriggerCancelledRuns` * `TriggerFailedRuns` * `TriggerSucceededRuns` ## microsoft.datalakeanalytics/accounts * `JobAUEndedCancelled` * `JobAUEndedFailure` * `JobAUEndedSuccess` * `JobEndedCancelled` * `JobEndedFailure` * `JobEndedSuccess` * `JobStage` ## microsoft.datalakestore/accounts * `DataRead` * `DataWritten` * `ReadRequests` * `TotalStorage` * `WriteRequests` ## microsoft.dataprotection/backupvaults * `BackupHealthEvent` * `RestoreHealthEvent` ## microsoft.dbformariadb/servers * `active_connections` * `backup_storage_used` * `connections_failed` * `cpu_percent` * `io_consumption_percent` * `memory_percent` * `network_bytes_egress` * `network_bytes_ingress` * `seconds_behind_master` * `serverlog_storage_limit` * `serverlog_storage_percent` * `serverlog_storage_usage` * `storage_limit` * `storage_percent` * `storage_used` ## microsoft.dbformysql/flexibleservers * `Com_alter_table` * `Com_create_db` * `Com_create_table` * `Com_delete` * `Com_drop_db` * `Com_drop_table` * `Com_insert` * `Com_select` * `Com_update` * `HA_IO_status` * `HA_SQL_status` * `HA_replication_lag` * `Innodb_buffer_pool_pages_data` * `Innodb_buffer_pool_pages_dirty` * `Innodb_buffer_pool_pages_flushed` * `Innodb_buffer_pool_pages_free` * `Innodb_buffer_pool_read_requests` * `Innodb_buffer_pool_reads` * `Innodb_data_writes` * `Innodb_row_lock_time` * `Innodb_row_lock_waits` * `Queries` * `Replica_IO_Running` * `Replica_SQL_Running` * `Slow_queries` * `Sort_merge_passes` * `Threads_running` * `Uptime` * `aborted_connections` * `active_connections` * `active_transactions` * `backup_storage_used` * `binlog_storage_used` * `cpu_credits_consumed` * `cpu_credits_remaining` * `cpu_percent` * `data_storage_used` * `ibdata1_storage_used` * `io_consumption_percent` * `lock_deadlocks` * `lock_timeouts` * `memory_percent` * `network_bytes_egress` * `network_bytes_ingress` * `others_storage_used` * `replication_lag` * `serverlog_storage_limit` * `serverlog_storage_percent` * `serverlog_storage_usage` * `storage_io_count` * `storage_limit` * `storage_percent` * `storage_used` * `total_connections` * `trx_rseg_history_len` ## microsoft.dbformysql/servers * `active_connections` * `backup_storage_used` * `connections_failed` * `cpu_percent` * `io_consumption_percent` * `memory_percent` * `network_bytes_egress` * `network_bytes_ingress` * `seconds_behind_master` * `serverlog_storage_limit` * `serverlog_storage_percent` * `serverlog_storage_usage` * `storage_limit` * `storage_percent` * `storage_used` ## microsoft.dbforpostgresql/flexibleservers * `active_connections` * `analyze_count_user_tables` * `autoanalyze_count_user_tables` * `autovacuum_count_user_tables` * `backup_storage_used` * `blks_hit` * `blks_read` * `bloat_percent` * `client_connections_active` * `client_connections_waiting` * `connections_failed` * `connections_succeeded` * `cpu_credits_consumed` * `cpu_credits_remaining` * `cpu_percent` * `database_size_bytes` * `deadlocks` * `disk_bandwidth_consumed_percentage` * `disk_iops_consumed_percentage` * `disk_queue_depth` * `iops` * `is_db_alive` * `logical_replication_delay_in_bytes` * `longest_query_time_sec` * `longest_transaction_time_sec` * `max_connections` * `maximum_used_transactionIDs` * `memory_percent` * `n_dead_tup_user_tables` * `n_live_tup_user_tables` * `n_mod_since_analyze_user_tables` * `network_bytes_egress` * `network_bytes_ingress` * `num_pools` * `numbackends` * `oldest_backend_time_sec` * `oldest_backend_xmin` * `oldest_backend_xmin_age` * `physical_replication_delay_in_bytes` * `physical_replication_delay_in_seconds` * `postmaster_process_cpu_usage_percent` * `read_iops` * `read_throughput` * `server_connections_active` * `server_connections_idle` * `sessions_by_state` * `sessions_by_wait_event_type` * `storage_free` * `storage_percent` * `storage_used` * `tables_analyzed_user_tables` * `tables_autoanalyzed_user_tables` * `tables_autovacuumed_user_tables` * `tables_counter_user_tables` * `tables_vacuumed_user_tables` * `tcp_connection_backlog` * `temp_bytes` * `temp_files` * `total_pooled_connections` * `tps` * `tup_deleted` * `tup_fetched` * `tup_inserted` * `tup_returned` * `tup_updated` * `txlogs_storage_used` * `vacuum_count_user_tables` * `write_iops` * `write_throughput` * `xact_commit` * `xact_rollback` * `xact_total` ## microsoft.dbforpostgresql/servergroupsv2 * `active_connections` * `apps_reserved_memory_percent` * `cpu_credits_consumed` * `cpu_credits_remaining` * `cpu_percent` * `iops` * `memory_percent` * `network_bytes_egress` * `network_bytes_ingress` * `replication_lag` * `storage_percent` * `storage_used` * `vm_cached_bandwidth_percent` * `vm_cached_iops_percent` * `vm_uncached_bandwidth_percent` * `vm_uncached_iops_percent` ## microsoft.dbforpostgresql/servers * `active_connections` * `backup_storage_used` * `connections_failed` * `cpu_percent` * `io_consumption_percent` * `memory_percent` * `network_bytes_egress` * `network_bytes_ingress` * `pg_replica_log_delay_in_bytes` * `pg_replica_log_delay_in_seconds` * `serverlog_storage_limit` * `serverlog_storage_percent` * `serverlog_storage_usage` * `storage_limit` * `storage_percent` * `storage_used` ## microsoft.dbforpostgresql/serversv2 * `active_connections` * `cpu_percent` * `iops` * `memory_percent` * `network_bytes_egress` * `network_bytes_ingress` * `storage_percent` * `storage_used` ## microsoft.devices/iothubs * `C2DMessagesExpired` * `EventGridDeliveries` * `EventGridLatency` * `RoutingDataSizeInBytesDelivered` * `RoutingDeliveries` * `RoutingDeliveryLatency` * `c2d.commands.egress.abandon.success` * `c2d.commands.egress.complete.success` * `c2d.commands.egress.reject.success` * `c2d.methods.failure` * `c2d.methods.requestSize` * `c2d.methods.responseSize` * `c2d.methods.success` * `c2d.twin.read.failure` * `c2d.twin.read.size` * `c2d.twin.read.success` * `c2d.twin.update.failure` * `c2d.twin.update.size` * `c2d.twin.update.success` * `configurations` * `connect.success` * `connectedDeviceCount` * `d2c.endpoints.egress.builtIn.events` * `d2c.endpoints.egress.eventHubs` * `d2c.endpoints.egress.serviceBusQueues` * `d2c.endpoints.egress.serviceBusTopics` * `d2c.endpoints.egress.storage` * `d2c.endpoints.egress.storage.blobs` * `d2c.endpoints.egress.storage.bytes` * `d2c.endpoints.latency.builtIn.events` * `d2c.endpoints.latency.eventHubs` * `d2c.endpoints.latency.serviceBusQueues` * `d2c.endpoints.latency.serviceBusTopics` * `d2c.endpoints.latency.storage` * `d2c.telemetry.egress.dropped` * `d2c.telemetry.egress.fallback` * `d2c.telemetry.egress.invalid` * `d2c.telemetry.egress.orphaned` * `d2c.telemetry.egress.success` * `d2c.telemetry.ingress.allProtocol` * `d2c.telemetry.ingress.sendThrottle` * `d2c.telemetry.ingress.success` * `d2c.twin.read.failure` * `d2c.twin.read.size` * `d2c.twin.read.success` * `d2c.twin.update.failure` * `d2c.twin.update.size` * `d2c.twin.update.success` * `dailyMessageQuotaUsed` * `deviceDataUsage` * `deviceDataUsageV2` * `devices.connectedDevices.allProtocol` * `devices.totalDevices` * `jobs.cancelJob.failure` * `jobs.cancelJob.success` * `jobs.completed` * `jobs.createDirectMethodJob.failure` * `jobs.createDirectMethodJob.success` * `jobs.createTwinUpdateJob.failure` * `jobs.createTwinUpdateJob.success` * `jobs.failed` * `jobs.listJobs.failure` * `jobs.listJobs.success` * `jobs.queryJobs.failure` * `jobs.queryJobs.success` * `totalDeviceCount` * `twinQueries.failure` * `twinQueries.resultSize` * `twinQueries.success` ## microsoft.devices/provisioningservices * `AttestationAttempts` * `DeviceAssignments` * `RegistrationAttempts` ## microsoft.documentdb/cassandraclusters * `cassandra_cache_capacity` * `cassandra_cache_entries` * `cassandra_cache_hit_rate` * `cassandra_cache_hits` * `cassandra_cache_miss_latency_histogram` * `cassandra_cache_miss_latency_p99` * `cassandra_cache_requests` * `cassandra_cache_size` * `cassandra_client_auth_failure` * `cassandra_client_auth_failure2` * `cassandra_client_auth_failure3` * `cassandra_client_auth_success` * `cassandra_client_connected_native_clients` * `cassandra_client_request_condition_not_met` * `cassandra_client_request_contention_histogram` * `cassandra_client_request_contention_histogram_p99` * `cassandra_client_request_failures` * `cassandra_client_request_failures2` * `cassandra_client_request_failures3` * `cassandra_client_request_latency_histogram` * `cassandra_client_request_latency_max` * `cassandra_client_request_latency_p99` * `cassandra_client_request_timeouts` * `cassandra_client_request_timeouts2` * `cassandra_client_request_timeouts3` * `cassandra_client_request_unfinished_commit` * `cassandra_commit_log_waiting_on_commit_latency_histogram` * `cassandra_cql_prepared_statements_executed` * `cassandra_cql_regular_statements_executed` * `cassandra_dropped_message_count` * `cassandra_dropped_message_cross_node_latency` * `cassandra_dropped_message_cross_node_latency_p99` * `cassandra_dropped_message_internal_latency` * `cassandra_dropped_message_rate` * `cassandra_hints_failed_rate` * `cassandra_hints_succeeded_rate` * `cassandra_hints_timed_out_rate` * `cassandra_jvm_gc_count` * `cassandra_jvm_gc_time` * `cassandra_storage_total_hints_counter_total` * `cassandra_storage_total_hints_in_progress_counter_total` * `cassandra_table_all_memtables_live_data_size` * `cassandra_table_all_memtables_off_heap_size` * `cassandra_table_bloom_filter_disk_space_used` * `cassandra_table_bloom_filter_false_positives` * `cassandra_table_bloom_filter_false_ratio` * `cassandra_table_bloom_filter_off_heap_memory_used` * `cassandra_table_bytes_flushed` * `cassandra_table_cas_commit` * `cassandra_table_cas_commit_p99` * `cassandra_table_cas_prepare` * `cassandra_table_cas_prepare_p99` * `cassandra_table_cas_propose` * `cassandra_table_cas_propose_p99` * `cassandra_table_col_update_time_delta_histogram` * `cassandra_table_col_update_time_delta_histogram_p99` * `cassandra_table_compaction_bytes_written` * `cassandra_table_compression_metadata_off_heap_memory_used` * `cassandra_table_compression_ratio` * `cassandra_table_coordinator_read_latency` * `cassandra_table_coordinator_read_latency_p99` * `cassandra_table_coordinator_scan_latency` * `cassandra_table_coordinator_scan_latency_p99` * `cassandra_table_dropped_mutations` * `cassandra_table_dropped_mutations2` * `cassandra_table_dropped_mutations3` * `cassandra_table_estimated_column_count_histogram` * `cassandra_table_estimated_column_count_histogram_p99` * `cassandra_table_estimated_partition_count` * `cassandra_table_estimated_partition_size_histogram` * `cassandra_table_estimated_partition_size_histogram_p99` * `cassandra_table_index_summary_off_heap_memory_used` * `cassandra_table_key_cache_hit_rate` * `cassandra_table_live_disk_space_used` * `cassandra_table_live_scanned_histogram` * `cassandra_table_live_scanned_histogram_p99` * `cassandra_table_live_sstable_count` * `cassandra_table_max_partition_size` * `cassandra_table_mean_partition_size` * `cassandra_table_memtable_columns_count` * `cassandra_table_memtable_off_heap_size` * `cassandra_table_memtable_on_heap_size` * `cassandra_table_memtable_switch_count` * `cassandra_table_min_partition_size` * `cassandra_table_pending_compactions` * `cassandra_table_pending_compactions2` * `cassandra_table_pending_compactions3` * `cassandra_table_pending_flushes` * `cassandra_table_pending_flushes2` * `cassandra_table_pending_flushes3` * `cassandra_table_percent_repaired` * `cassandra_table_range_latency` * `cassandra_table_range_latency_p99` * `cassandra_table_read_latency` * `cassandra_table_read_latency_p99` * `cassandra_table_row_cache_hit` * `cassandra_table_row_cache_hit_out_of_range` * `cassandra_table_row_cache_miss` * `cassandra_table_speculative_retries` * `cassandra_table_sstables_per_read_histogram` * `cassandra_table_sstables_per_read_histogram_p99` * `cassandra_table_tombstone_scanned_histogram` * `cassandra_table_tombstone_scanned_histogram_p99` * `cassandra_table_total_disk_space_used` * `cassandra_table_total_disk_space_used2` * `cassandra_table_total_disk_space_used3` * `cassandra_table_view_lock_acquire_time` * `cassandra_table_view_lock_acquire_time_p99` * `cassandra_table_view_read_time` * `cassandra_table_view_read_time_p99` * `cassandra_table_waiting_on_free_memtable_space` * `cassandra_table_waiting_on_free_memtable_space_p99` * `cassandra_table_write_latency` * `cassandra_table_write_latency_p99` * `cassandra_thread_pools_active_tasks` * `cassandra_thread_pools_currently_blocked_tasks` * `cassandra_thread_pools_currently_blocked_tasks2` * `cassandra_thread_pools_currently_blocked_tasks3` * `cassandra_thread_pools_max_pool_size` * `cassandra_thread_pools_pending_tasks` * `cassandra_thread_pools_total_blocked_tasks` * `cpu` * `disk_utilization` * `diskio_merged_reads` * `diskio_merged_writes` * `diskio_read_bytes` * `diskio_read_time` * `diskio_reads` * `diskio_write_bytes` * `diskio_write_time` * `diskio_writes` * `ethtool_rx_bytes` * `ethtool_rx_packets` * `ethtool_tx_bytes` * `ethtool_tx_packets` * `iops` * `percent_mem` * `raid_array_degraded` * `raid_array_rebuild` * `total_cpu` ## microsoft.documentdb/databaseaccounts * `AddRegion` * `AutoscaleMaxThroughput` * `AutoscaledRU` * `AvailableStorage` * `BackupModeUpdate` * `BackupPolicyIntervalUpdate` * `BackupPolicyRedundancyUpdate` * `BackupPolicyRetentionUpdate` * `BackupPolicyUpdate` * `CassandraConnectionClosures` * `CassandraConnectorAvgReplicationLatency` * `CassandraConnectorReplicationHealthStatus` * `CassandraKeyspaceCreate` * `CassandraKeyspaceDelete` * `CassandraKeyspaceThroughputUpdate` * `CassandraKeyspaceUpdate` * `CassandraRequestCharges` * `CassandraRequests` * `CassandraTableCreate` * `CassandraTableDelete` * `CassandraTableThroughputUpdate` * `CassandraTableUpdate` * `ContinuousTierUpdate` * `CreateAccount` * `DataUsage` * `DedicatedGatewayAverageCPUUsage` * `DedicatedGatewayAverageMemoryUsage` * `DedicatedGatewayCPUUsage` * `DedicatedGatewayMaximumCPUUsage` * `DedicatedGatewayMemoryUsage` * `DedicatedGatewayRequests` * `DeleteAccount` * `DocumentCount` * `DocumentQuota` * `GlobalSecondaryIndexCatchupGapInMinutes` * `GlobalSecondaryIndexPropagationLatencyInSeconds` * `GremlinDatabaseCreate` * `GremlinDatabaseDelete` * `GremlinDatabaseThroughputUpdate` * `GremlinDatabaseUpdate` * `GremlinGraphCreate` * `GremlinGraphDelete` * `GremlinGraphThroughputUpdate` * `GremlinGraphUpdate` * `GremlinRequestCharges` * `GremlinRequests` * `IndexUsage` * `IntegratedCacheEvictedEntriesSize` * `IntegratedCacheItemExpirationCount` * `IntegratedCacheItemHitRate` * `IntegratedCacheQueryExpirationCount` * `IntegratedCacheQueryHitRate` * `MaterializedViewCatchupGapInMinutes` * `MaterializedViewsBuilderAverageCPUUsage` * `MaterializedViewsBuilderAverageMemoryUsage` * `MaterializedViewsBuilderMaximumCPUUsage` * `MetadataRequests` * `MongoCollectionCreate` * `MongoCollectionDelete` * `MongoCollectionThroughputUpdate` * `MongoCollectionUpdate` * `MongoDBDatabaseCreate` * `MongoDBDatabaseUpdate` * `MongoDatabaseDelete` * `MongoDatabaseThroughputUpdate` * `MongoRequestCharge` * `MongoRequests` * `NormalizedRUConsumption` * `OfflineRegion` * `OnlineRegion` * `PhysicalPartitionCount` * `PhysicalPartitionSizeInfo` * `PhysicalPartitionThroughputInfo` * `ProvisionedThroughput` * `RegionFailover` * `RemoveRegion` * `ReplicationLatency` * `ServerSideLatency` * `ServerSideLatencyDirect` * `ServerSideLatencyGateway` * `ServiceAvailability` * `SqlContainerCreate` * `SqlContainerDelete` * `SqlContainerThroughputUpdate` * `SqlContainerUpdate` * `SqlDatabaseCreate` * `SqlDatabaseDelete` * `SqlDatabaseThroughputUpdate` * `SqlDatabaseUpdate` * `TableTableCreate` * `TableTableDelete` * `TableTableThroughputUpdate` * `TableTableUpdate` * `TotalRequestUnits` * `TotalRequestUnitsPreview` * `TotalRequests` * `TotalRequestsPreview` * `UpdateAccountKeys` * `UpdateAccountNetworkSettings` * `UpdateAccountReplicationSettings` * `UpdateDiagnosticsSettings` ## microsoft.eventgrid/domains * `AdvancedFilterEvaluationCount` * `DeadLetteredCount` * `DeliveryAttemptFailCount` * `DeliverySuccessCount` * `DestinationProcessingDurationInMs` * `DroppedEventCount` * `MatchedEventCount` * `PublishFailCount` * `PublishSuccessCount` * `PublishSuccessLatencyInMs` ## microsoft.eventgrid/eventsubscriptions * `DeadLetteredCount` * `DeliveryAttemptFailCount` * `DeliverySuccessCount` * `DestinationProcessingDurationInMs` * `DroppedEventCount` * `MatchedEventCount` ## microsoft.eventgrid/extensiontopics * `PublishFailCount` * `PublishSuccessCount` * `PublishSuccessLatencyInMs` * `UnmatchedEventCount` ## microsoft.eventgrid/partnernamespaces * `PublishFailCount` * `PublishSuccessCount` * `PublishSuccessLatencyInMs` * `UnmatchedEventCount` ## microsoft.eventgrid/partnertopics * `AdvancedFilterEvaluationCount` * `DeadLetteredCount` * `DeliveryAttemptFailCount` * `DeliverySuccessCount` * `DestinationProcessingDurationInMs` * `DroppedEventCount` * `MatchedEventCount` * `PublishSuccessCount` * `UnmatchedEventCount` ## microsoft.eventgrid/systemtopics * `AdvancedFilterEvaluationCount` * `DeadLetteredCount` * `DeliveryAttemptFailCount` * `DeliverySuccessCount` * `DestinationProcessingDurationInMs` * `DroppedEventCount` * `MatchedEventCount` * `PublishFailCount` * `PublishSuccessCount` * `PublishSuccessLatencyInMs` * `ServerDeliverySuccessRate` * `UnmatchedEventCount` ## microsoft.eventgrid/topics * `AdvancedFilterEvaluationCount` * `DeadLetteredCount` * `DeliveryAttemptFailCount` * `DeliverySuccessCount` * `DestinationProcessingDurationInMs` * `DroppedEventCount` * `MatchedEventCount` * `PublishFailCount` * `PublishSuccessCount` * `PublishSuccessLatencyInMs` * `UnmatchedEventCount` ## microsoft.eventhub/clusters * `ActiveConnections` * `AvailableMemory` * `CPU` * `CaptureBacklog` * `CapturedBytes` * `CapturedMessages` * `ClusterUtilization` * `ConnectionsClosed` * `ConnectionsOpened` * `IncomingBytes` * `IncomingMessages` * `IncomingRequests` * `OutgoingBytes` * `OutgoingMessages` * `QuotaExceededErrors` * `ServerErrors` * `Size` * `SuccessfulRequests` * `ThrottledRequests` * `UserErrors` ## microsoft.eventhub/namespaces * `ActiveConnections` * `CaptureBacklog` * `CapturedBytes` * `CapturedMessages` * `ConnectionsClosed` * `ConnectionsOpened` * `EHABL` * `EHAMBS` * `EHAMSGS` * `EHINBYTES` * `EHINMBS` * `EHINMSGS` * `EHOUTBYTES` * `EHOUTMBS` * `EHOUTMSGS` * `FAILREQ` * `INMSGS` * `INREQS` * `INTERR` * `IncomingBytes` * `IncomingMessages` * `IncomingRequests` * `MISCERR` * `NamespaceCpuUsage` * `NamespaceMemoryUsage` * `OUTMSGS` * `OutgoingBytes` * `OutgoingMessages` * `QuotaExceededErrors` * `ReplicationLagCount` * `ReplicationLagDuration` * `SUCCREQ` * `SVRBSY` * `ServerErrors` * `Size` * `SuccessfulRequests` * `ThrottledRequests` * `UserErrors` ## microsoft.hdinsight/clusters * `CategorizedGatewayRequests` * `GatewayRequests` * `KafkaRestProxy.ConsumerRequest.m1_delta` * `KafkaRestProxy.ConsumerRequestFail.m1_delta` * `KafkaRestProxy.ConsumerRequestTime.p95` * `KafkaRestProxy.ConsumerRequestWaitingInQueueTime.p95` * `KafkaRestProxy.MessagesIn.m1_delta` * `KafkaRestProxy.MessagesOut.m1_delta` * `KafkaRestProxy.OpenConnections` * `KafkaRestProxy.ProducerRequest.m1_delta` * `KafkaRestProxy.ProducerRequestFail.m1_delta` * `KafkaRestProxy.ProducerRequestTime.p95` * `KafkaRestProxy.ProducerRequestWaitingInQueueTime.p95` * `NumActiveWorkers` * `PendingCPU` * `PendingMemory` ## microsoft.insights/datacollectionrules * `ApiCallReceived_Count` * `BytesReceived_Count` * `MetricIngestionRequest_Count` * `RowsDropped_Count` * `RowsReceived_Count` * `TransformationErrors_Count` * `TransformationRuntime_DurationMs` ## microsoft.keyvault/managedhsms * `Availability` * `ServiceApiHit` * `ServiceApiLatency` ## microsoft.keyvault/vaults * `Availability` * `SaturationShoebox` * `ServiceApiHit` * `ServiceApiLatency` * `ServiceApiResult` ## microsoft.kusto/clusters * `BatchBlobCount` * `BatchDuration` * `BatchSize` * `BatchesProcessed` * `BlobsDropped` * `BlobsProcessed` * `BlobsReceived` * `CPU` * `CacheUtilization` * `CacheUtilizationFactor` * `ContinuousExportMaxLatenessMinutes` * `ContinuousExportNumOfRecordsExported` * `ContinuousExportPendingCount` * `ContinuousExportResult` * `DiscoveryLatency` * `EventsDropped` * `EventsProcessed` * `EventsReceived` * `ExportUtilization` * `FollowerLatency` * `IngestionLatencyInSeconds` * `IngestionResult` * `IngestionUtilization` * `IngestionVolumeInMB` * `InstanceCount` * `KeepAlive` * `MaterializedViewAgeMinutes` * `MaterializedViewAgeSeconds` * `MaterializedViewDataLoss` * `MaterializedViewExtentsRebuild` * `MaterializedViewHealth` * `MaterializedViewRecordsInDelta` * `MaterializedViewResult` * `PartitioningPercentage` * `PartitioningPercentageHot` * `ProcessedPartitionedRecords` * `QueryDuration` * `QueryResult` * `QueueLength` * `QueueOldestMessage` * `ReceivedDataSizeBytes` * `StageLatency` * `StreamingIngestDataRate` * `StreamingIngestDuration` * `StreamingIngestResults` * `StreamingIngestUtilization` * `TotalNumberOfConcurrentQueries` * `TotalNumberOfExtents` * `TotalNumberOfThrottledCommands` * `TotalNumberOfThrottledQueries` * `WeakConsistencyLatency` ## microsoft.logic/workflows * `ActionLatency` * `ActionSuccessLatency` * `ActionThrottledEvents` * `ActionsCompleted` * `ActionsFailed` * `ActionsSkipped` * `ActionsStarted` * `ActionsSucceeded` * `AgentLoopExecution` * `BillableActionExecutions` * `BillableTriggerExecutions` * `BillingUsageNativeOperation` * `BillingUsageStandardConnector` * `BillingUsageStorageConsumption` * `CompletionTokenOverflowUsage` * `PromptTokenOverflowUsage` * `RunFailurePercentage` * `RunLatency` * `RunStartThrottledEvents` * `RunSuccessLatency` * `RunThrottledEvents` * `RunsCancelled` * `RunsCompleted` * `RunsFailed` * `RunsStarted` * `RunsSucceeded` * `TotalBillableExecutions` * `TriggerFireLatency` * `TriggerLatency` * `TriggerSuccessLatency` * `TriggerThrottledEvents` * `TriggersCompleted` * `TriggersFailed` * `TriggersFired` * `TriggersSkipped` * `TriggersStarted` * `TriggersSucceeded` ## microsoft.machinelearningservices/workspaces * `Active Cores` * `Active Nodes` * `Agents` * `Cancel Requested Runs` * `Cancelled Runs` * `Completed Runs` * `CpuCapacityMillicores` * `CpuMemoryCapacityMegabytes` * `CpuMemoryUtilizationMegabytes` * `CpuMemoryUtilizationPercentage` * `CpuUtilization` * `CpuUtilizationMillicores` * `CpuUtilizationPercentage` * `DiskAvailMegabytes` * `DiskReadMegabytes` * `DiskUsedMegabytes` * `DiskWriteMegabytes` * `Errors` * `Failed Runs` * `Finalizing Runs` * `GpuCapacityMilliGPUs` * `GpuEnergyJoules` * `GpuMemoryCapacityMegabytes` * `GpuMemoryUtilization` * `GpuMemoryUtilizationMegabytes` * `GpuMemoryUtilizationPercentage` * `GpuUtilization` * `GpuUtilizationMilliGPUs` * `GpuUtilizationPercentage` * `IBReceiveMegabytes` * `IBTransmitMegabytes` * `Idle Cores` * `Idle Nodes` * `IndexedFiles` * `Leaving Cores` * `Leaving Nodes` * `Messages` * `Model Deploy Failed` * `Model Deploy Started` * `Model Deploy Succeeded` * `Model Register Failed` * `Model Register Succeeded` * `NetworkInputMegabytes` * `NetworkOutputMegabytes` * `Not Responding Runs` * `Not Started Runs` * `Preempted Cores` * `Preempted Nodes` * `Preparing Runs` * `Provisioning Runs` * `Queued Runs` * `Quota Utilization Percentage` * `Runs` * `Started Runs` * `Starting Runs` * `StorageAPIFailureCount` * `StorageAPISuccessCount` * `Threads` * `Tokens` * `ToolCalls` * `Total Cores` * `Total Nodes` * `Unusable Cores` * `Unusable Nodes` * `Warnings` ## microsoft.monitor/accounts * `ActiveTimeSeries` * `ActiveTimeSeriesLimit` * `ActiveTimeSeriesPercentUtilization` * `EventsDropped` * `EventsPerMinuteIngested` * `EventsPerMinuteIngestedLimit` * `EventsPerMinuteIngestedPercentUtilization` * `TimeSeriesSamplesDropped` ## microsoft.network/applicationgateways * `ApplicationGatewayTotalTime` * `AvgRequestCountPerHealthyHost` * `AzWAFCaptchaChallengeRequestCount` * `AzWAFJSChallengeRequestCount` * `AzwafBotProtection` * `AzwafCustomRule` * `AzwafPenaltyBoxHits` * `AzwafPenaltyBoxSize` * `AzwafSecRule` * `AzwafTotalRequests` * `BackendConnectTime` * `BackendFirstByteResponseTime` * `BackendLastByteResponseTime` * `BackendResponseStatus` * `BlockedCount` * `BytesReceived` * `BytesSent` * `CapacityUnits` * `ClientRtt` * `ComputeUnits` * `CpuUtilization` * `CurrentConnections` * `EstimatedBilledCapacityUnits` * `FailedRequests` * `FixedBillableCapacityUnits` * `HealthyHostCount` * `MatchedCount` * `NewConnectionsPerSecond` * `ResponseStatus` * `Throughput` * `TlsProtocol` * `TotalRequests` * `UnhealthyHostCount` * `WebSocketActiveConnections` * `WebsocketSpecificCloseStatusCode` ## microsoft.network/azurefirewalls * `ApplicationRuleHit` * `DataProcessed` * `FirewallHealth` * `FirewallLatencyPng` * `NetworkRuleHit` * `ObservedCapacity` * `SNATPortUtilization` * `Throughput` ## microsoft.network/expressroutecircuits * `ArpAvailability` * `BgpAvailability` * `BitsInPerSecond` * `BitsOutPerSecond` * `EgressBandwidthUtilization` * `FastPathRoutesCountForCircuit` * `GlobalReachBitsInPerSecond` * `GlobalReachBitsOutPerSecond` * `IngressBandwidthUtilization` * `QosDropBitsInPerSecond` * `QosDropBitsOutPerSecond` ## microsoft.network/expressroutecircuits/peerings * `BitsInPerSecond` * `BitsOutPerSecond` ## microsoft.network/expressroutegateways * `ErGatewayConnectionBitsInPerSecond` * `ErGatewayConnectionBitsOutPerSecond` * `ExpressRouteGatewayActiveFlows` * `ExpressRouteGatewayBitsPerSecond` * `ExpressRouteGatewayCountOfRoutesAdvertisedToPeer` * `ExpressRouteGatewayCountOfRoutesLearnedFromPeer` * `ExpressRouteGatewayCpuUtilization` * `ExpressRouteGatewayFrequencyOfRoutesChanged` * `ExpressRouteGatewayMaxFlowsCreationRate` * `ExpressRouteGatewayNumberOfVmInVnet` * `ExpressRouteGatewayPacketsPerSecond` ## microsoft.network/expressrouteports * `AdminState` * `FastPathRoutesCountForDirectPort` * `LineProtocol` * `PortBitsInPerSecond` * `PortBitsOutPerSecond` * `RxLightLevel` * `TxLightLevel` ## microsoft.network/frontdoors * `BackendHealthPercentage` * `BackendRequestCount` * `BackendRequestLatency` * `BillableResponseSize` * `RequestCount` * `RequestSize` * `ResponseSize` * `TotalLatency` * `WebApplicationFirewallRequestCount` ## microsoft.network/loadbalancers * `AllocatedSnatPorts` * `ByteCount` * `DipAvailability` * `GlobalBackendAvailability` * `PacketCount` * `SYNCount` * `SnatConnectionCount` * `UsedSnatPorts` * `VipAvailability` ## microsoft.network/natgateways * `ByteCount` * `DatapathAvailability` * `PacketCount` * `PacketDropCount` * `SNATConnectionCount` * `TotalConnectionCount` ## microsoft.network/networkinterfaces * `BytesReceivedRate` * `BytesSentRate` * `PacketsReceivedRate` * `PacketsSentRate` ## microsoft.network/p2svpngateways * `P2SBandwidth` * `P2SConnectionCount` * `UserVpnRouteCount` ## microsoft.network/privateendpoints * `PEBytesIn` * `PEBytesOut` ## microsoft.network/privatelinkservices * `PLSBytesIn` * `PLSBytesOut` * `PLSNatPortsUsage` ## microsoft.network/publicipaddresses * `ByteCount` * `BytesDroppedDDoS` * `BytesForwardedDDoS` * `BytesInDDoS` * `DDoSTriggerSYNPackets` * `DDoSTriggerTCPPackets` * `DDoSTriggerUDPPackets` * `IfUnderDDoSAttack` * `PacketCount` * `PacketsDroppedDDoS` * `PacketsForwardedDDoS` * `PacketsInDDoS` * `SynCount` * `TCPBytesDroppedDDoS` * `TCPBytesForwardedDDoS` * `TCPBytesInDDoS` * `TCPPacketsDroppedDDoS` * `TCPPacketsForwardedDDoS` * `TCPPacketsInDDoS` * `UDPBytesDroppedDDoS` * `UDPBytesForwardedDDoS` * `UDPBytesInDDoS` * `UDPPacketsDroppedDDoS` * `UDPPacketsForwardedDDoS` * `UDPPacketsInDDoS` * `VipAvailability` ## microsoft.network/virtualhubs * `BgpPeerStatus` * `CountOfRoutesAdvertisedToPeer` * `CountOfRoutesLearnedFromPeer` * `RoutingInfrastructureUnits` * `SpokeVMUtilization` * `VirtualHubDataProcessed` ## microsoft.network/virtualnetworkappliances * `BytesReceived` * `BytesSent` * `CreationRateMaxTotalFlowsIn` * `CreationRateMaxTotalFlowsOut` * `CurrentTotalFlowsIn` * `CurrentTotalFlowsOut` * `PacketsReceived` * `PacketsSent` ## microsoft.network/virtualnetworkgateways * `AverageBandwidth` * `BgpPeerStatus` * `BgpRoutesAdvertised` * `BgpRoutesLearned` * `ExpressRouteGatewayActiveFlows` * `ExpressRouteGatewayBitsPerSecond` * `ExpressRouteGatewayCountOfRoutesAdvertisedToPeer` * `ExpressRouteGatewayCountOfRoutesLearnedFromPeer` * `ExpressRouteGatewayCpuUtilization` * `ExpressRouteGatewayFrequencyOfRoutesChanged` * `ExpressRouteGatewayMaxFlowsCreationRate` * `ExpressRouteGatewayNumberOfVmInVnet` * `ExpressRouteGatewayPacketsPerSecond` * `InboundFlowsCount` * `MmsaCount` * `OutboundFlowsCount` * `P2SBandwidth` * `P2SConnectionCount` * `QmsaCount` * `ScalableExpressRouteGatewayActiveFlows` * `ScalableExpressRouteGatewayBitsPerSecond` * `ScalableExpressRouteGatewayCountOfRoutesAdvertisedToPeer` * `ScalableExpressRouteGatewayCountOfRoutesLearnedFromPeer` * `ScalableExpressRouteGatewayCpuUtilization` * `ScalableExpressRouteGatewayFrequencyOfRoutesChanged` * `ScalableExpressRouteGatewayMaxFlowsCreationRate` * `ScalableExpressRouteGatewayNumberOfVmInVnet` * `ScalableExpressRouteGatewayPacketsPerSecond` * `ScalableExpressRouteGatewayScaleUnit` * `TunnelAverageBandwidth` * `TunnelEgressBytes` * `TunnelEgressPacketDropCount` * `TunnelEgressPacketDropTSMismatch` * `TunnelEgressPackets` * `TunnelIngressBytes` * `TunnelIngressPacketDropCount` * `TunnelIngressPacketDropTSMismatch` * `TunnelIngressPackets` * `TunnelNatAllocations` * `TunnelNatFlowCount` * `TunnelNatPacketDrop` * `TunnelNatedBytes` * `TunnelNatedPackets` * `TunnelPeakPackets` * `TunnelReverseNatedBytes` * `TunnelReverseNatedPackets` * `TunnelTotalFlowCount` * `UserVpnRouteCount` * `VnetAddressPrefixCount` ## microsoft.network/virtualnetworks * `BytesDroppedDDoS` * `BytesForwardedDDoS` * `BytesInDDoS` * `DDoSTriggerSYNPackets` * `DDoSTriggerTCPPackets` * `DDoSTriggerUDPPackets` * `IfUnderDDoSAttack` * `PacketsDroppedDDoS` * `PacketsForwardedDDoS` * `PacketsInDDoS` * `PingMeshAverageRoundtripMs` * `PingMeshProbesFailedPercent` * `TCPBytesDroppedDDoS` * `TCPBytesForwardedDDoS` * `TCPBytesInDDoS` * `TCPPacketsDroppedDDoS` * `TCPPacketsForwardedDDoS` * `TCPPacketsInDDoS` * `UDPBytesDroppedDDoS` * `UDPBytesForwardedDDoS` * `UDPBytesInDDoS` * `UDPPacketsDroppedDDoS` * `UDPPacketsForwardedDDoS` * `UDPPacketsInDDoS` ## microsoft.network/vpngateways * `AverageBandwidth` * `BgpPeerStatus` * `BgpRoutesAdvertised` * `BgpRoutesLearned` * `InboundFlowsCount` * `MmsaCount` * `OutboundFlowsCount` * `QmsaCount` * `TunnelAverageBandwidth` * `TunnelEgressBytes` * `TunnelEgressPacketDropCount` * `TunnelEgressPacketDropTSMismatch` * `TunnelEgressPackets` * `TunnelIngressBytes` * `TunnelIngressPacketDropCount` * `TunnelIngressPacketDropTSMismatch` * `TunnelIngressPackets` * `TunnelNatAllocations` * `TunnelNatFlowCount` * `TunnelNatPacketDrop` * `TunnelNatedBytes` * `TunnelNatedPackets` * `TunnelPeakPackets` * `TunnelReverseNatedBytes` * `TunnelReverseNatedPackets` * `TunnelTotalFlowCount` * `VnetAddressPrefixCount` ## microsoft.notificationhubs/namespaces/notificationhubs * `incoming` * `incoming.all.failedrequests` * `incoming.all.requests` * `incoming.scheduled` * `incoming.scheduled.cancel` * `installation.all` * `installation.delete` * `installation.get` * `installation.patch` * `installation.upsert` * `notificationhub.pushes` * `outgoing.allpns.badorexpiredchannel` * `outgoing.allpns.channelerror` * `outgoing.allpns.invalidpayload` * `outgoing.allpns.pnserror` * `outgoing.allpns.success` * `outgoing.apns.badchannel` * `outgoing.apns.expiredchannel` * `outgoing.apns.invalidcredentials` * `outgoing.apns.invalidnotificationsize` * `outgoing.apns.pnserror` * `outgoing.apns.success` * `outgoing.fcmv1.badchannel` * `outgoing.fcmv1.invalidcredentials` * `outgoing.fcmv1.invalidnotificationformat` * `outgoing.fcmv1.invalidnotificationsize` * `outgoing.fcmv1.pnserror` * `outgoing.fcmv1.success` * `outgoing.fcmv1.throttled` * `outgoing.fcmv1.wrongchannel` * `outgoing.gcm.authenticationerror` * `outgoing.gcm.badchannel` * `outgoing.gcm.expiredchannel` * `outgoing.gcm.invalidcredentials` * `outgoing.gcm.invalidnotificationformat` * `outgoing.gcm.invalidnotificationsize` * `outgoing.gcm.pnserror` * `outgoing.gcm.success` * `outgoing.gcm.throttled` * `outgoing.gcm.wrongchannel` * `outgoing.mpns.authenticationerror` * `outgoing.mpns.badchannel` * `outgoing.mpns.channeldisconnected` * `outgoing.mpns.dropped` * `outgoing.mpns.invalidcredentials` * `outgoing.mpns.invalidnotificationformat` * `outgoing.mpns.pnserror` * `outgoing.mpns.success` * `outgoing.mpns.throttled` * `outgoing.wns.authenticationerror` * `outgoing.wns.badchannel` * `outgoing.wns.channeldisconnected` * `outgoing.wns.channelthrottled` * `outgoing.wns.dropped` * `outgoing.wns.expiredchannel` * `outgoing.wns.invalidcredentials` * `outgoing.wns.invalidnotificationformat` * `outgoing.wns.invalidnotificationsize` * `outgoing.wns.invalidtoken` * `outgoing.wns.pnserror` * `outgoing.wns.success` * `outgoing.wns.throttled` * `outgoing.wns.tokenproviderunreachable` * `outgoing.wns.wrongtoken` * `registration.all` * `registration.create` * `registration.delete` * `registration.get` * `registration.update` * `scheduled.pending` ## microsoft.powerbidedicated/capacities * `cpu_metric` * `overload_metric` ## microsoft.recoveryservices/vaults * `BackupHealthEvent` * `RestoreHealthEvent` ## microsoft.relay/namespaces * `ActiveConnections` * `ActiveListeners` * `BytesTransferred` * `ListenerConnections-ClientError` * `ListenerConnections-ServerError` * `ListenerConnections-Success` * `ListenerConnections-TotalRequests` * `ListenerDisconnects` * `SenderConnections-ClientError` * `SenderConnections-ServerError` * `SenderConnections-Success` * `SenderConnections-TotalRequests` * `SenderDisconnects` ## microsoft.search/searchservices * `DocumentsProcessedCount` * `SearchLatency` * `SearchQueriesPerSecond` * `SkillExecutionCount` * `ThrottledSearchQueriesPercentage` ## microsoft.servicebus/namespaces * `AbandonMessage` * `ActiveConnections` * `ActiveMessages` * `CPUXNS` * `CompleteMessage` * `ConnectionsClosed` * `ConnectionsOpened` * `DeadletteredMessages` * `IncomingBytes` * `IncomingMessages` * `IncomingRequests` * `Messages` * `NamespaceCpuUsage` * `NamespaceMemoryUsage` * `OutgoingBytes` * `OutgoingMessages` * `PendingCheckpointOperationCount` * `ReplicationLagCount` * `ReplicationLagDuration` * `ScheduledMessages` * `ServerErrors` * `ServerSendLatency` * `Size` * `SuccessfulRequests` * `ThrottledRequests` * `UserErrors` * `WSXNS` ## microsoft.signalrservice/signalr * `ConnectionCloseCount` * `ConnectionCount` * `ConnectionOpenCount` * `ConnectionQuotaUtilization` * `InboundTraffic` * `MessageCount` * `OutboundTraffic` * `ServerConnectionLatency` * `ServerLoad` * `SystemErrors` * `UserErrors` ## microsoft.signalrservice/signalr/replicas * `ConnectionCloseCount` * `ConnectionCount` * `ConnectionOpenCount` * `ConnectionQuotaUtilization` * `InboundTraffic` * `MessageCount` * `OutboundTraffic` * `ServerConnectionLatency` * `ServerLoad` * `SystemErrors` * `UserErrors` ## microsoft.signalrservice/webpubsub * `ClientRequestStatus` * `ConnectionCloseCount` * `ConnectionOpenCount` * `ConnectionQuotaUtilization` * `InboundTraffic` * `OutboundTraffic` * `RestApiResponseTimeCount` * `ServerLoad` * `TotalConnectionCount` ## microsoft.signalrservice/webpubsub/replicas * `ClientRequestStatus` * `ConnectionCloseCount` * `ConnectionOpenCount` * `ConnectionQuotaUtilization` * `InboundTraffic` * `OutboundTraffic` * `RestApiResponseTimeCount` * `ServerLoad` * `TotalConnectionCount` ## microsoft.sql/managedinstances * `avg_cpu_percent` * `io_bytes_read` * `io_bytes_written` * `io_requests` * `reserved_storage_mb` * `storage_space_used_mb` * `virtual_core_count` ## microsoft.sql/servers/databases * `active_queries` * `allocated_data_storage` * `app_cpu_billed` * `app_cpu_billed_ha_replicas` * `app_cpu_percent` * `app_memory_percent` * `availability` * `base_blob_size_bytes` * `blocked_by_firewall` * `cache_hit_percent` * `cache_used_percent` * `connection_failed` * `connection_failed_user_error` * `connection_successful` * `cpu_limit` * `cpu_percent` * `cpu_used` * `deadlock` * `diff_backup_size_bytes` * `dtu_consumption_percent` * `dtu_limit` * `dtu_used` * `dwu_consumption_percent` * `dwu_limit` * `dwu_used` * `free_amount_consumed` * `free_amount_remaining` * `full_backup_size_bytes` * `ledger_digest_upload_failed` * `ledger_digest_upload_success` * `local_tempdb_usage_percent` * `log_backup_size_bytes` * `log_write_percent` * `memory_usage_percent` * `physical_data_read_percent` * `queued_queries` * `replication_lag_seconds` * `sessions_count` * `sessions_percent` * `snapshot_backup_size_bytes` * `sql_instance_cpu_percent` * `sql_instance_memory_percent` * `sqlserver_process_core_percent` * `sqlserver_process_memory_percent` * `storage` * `storage_percent` * `tempdb_data_size` * `tempdb_log_size` * `tempdb_log_used_percent` * `wlg_active_queries` * `wlg_active_queries_timeouts` * `wlg_allocation_relative_to_system_percent` * `wlg_allocation_relative_to_wlg_effective_cap_percent` * `wlg_effective_cap_resource_percent` * `wlg_effective_min_resource_percent` * `wlg_queued_queries` * `workers_percent` * `xtp_storage_percent` ## microsoft.sql/servers/elasticpools * `allocated_data_storage` * `allocated_data_storage_percent` * `app_cpu_billed` * `app_cpu_percent` * `app_memory_percent` * `cpu_limit` * `cpu_percent` * `cpu_used` * `dtu_consumption_percent` * `eDTU_limit` * `eDTU_used` * `log_write_percent` * `physical_data_read_percent` * `sessions_count` * `sessions_percent` * `sql_instance_cpu_percent` * `sql_instance_memory_percent` * `sqlserver_process_core_percent` * `sqlserver_process_memory_percent` * `storage_limit` * `storage_percent` * `storage_used` * `tempdb_data_size` * `tempdb_log_size` * `tempdb_log_used_percent` * `workers_percent` * `xtp_storage_percent` ## microsoft.storage/storageaccounts * `Availability` * `Egress` * `Egress-Priority` * `Ingress` * `Ingress-Priority` * `ReplicationLagSeconds` * `SuccessE2ELatency` * `SuccessServerLatency` * `Transactions` * `Transactions-Priority` * `UsedCapacity` ## microsoft.storage/storageaccounts/blobservices * `Availability` * `BlobCapacity` * `BlobCount` * `BlobProvisionedSize` * `ContainerCount` * `Egress` * `IndexCapacity` * `Ingress` * `SuccessE2ELatency` * `SuccessServerLatency` * `Transactions` ## microsoft.storage/storageaccounts/fileservices * `Availability` * `Egress` * `FileCapacity` * `FileCount` * `FileShareAvailableBurstCredits` * `FileShareCapacityQuota` * `FileShareCount` * `FileShareMaxUsedBandwidthMiBps` * `FileShareMaxUsedIOPS` * `FileSharePaidBurstingBandwidth` * `FileSharePaidBurstingIOS` * `FileShareProvisionedBandwidthMiBps` * `FileShareProvisionedIOPS` * `FileShareSnapshotCount` * `FileShareSnapshotSize` * `Ingress` * `PercentFileShareUtilization` * `SuccessE2ELatency` * `SuccessServerLatency` * `Transactions` ## microsoft.storage/storageaccounts/queueservices * `Availability` * `Egress` * `Ingress` * `QueueCapacity` * `QueueCount` * `QueueMessageCount` * `SuccessE2ELatency` * `SuccessServerLatency` * `Transactions` ## microsoft.storage/storageaccounts/tableservices * `Availability` * `Egress` * `Ingress` * `SuccessE2ELatency` * `SuccessServerLatency` * `TableCapacity` * `TableCount` * `TableEntityCount` * `Transactions` ## microsoft.streamanalytics/streamingjobs * `AMLCalloutFailedRequests` * `AMLCalloutInputEvents` * `AMLCalloutRequests` * `ConversionErrors` * `DeserializationError` * `DroppedOrAdjustedEvents` * `EarlyInputEvents` * `Errors` * `InputEventBytes` * `InputEvents` * `InputEventsSourcesBacklogged` * `InputEventsSourcesPerSecond` * `LateInputEvents` * `OutputEvents` * `OutputWatermarkDelaySeconds` * `ProcessCPUUsagePercentage` * `ResourceUtilization` ## microsoft.synapse/workspaces * `BuiltinSqlPoolDataProcessedBytes` * `BuiltinSqlPoolLoginAttempts` * `BuiltinSqlPoolRequestsEnded` * `IntegrationActivityRunsEnded` * `IntegrationLinkConnectionEvents` * `IntegrationLinkProcessedChangedRows` * `IntegrationLinkProcessedDataVolume` * `IntegrationLinkProcessingLatencyInSeconds` * `IntegrationLinkTableEvents` * `IntegrationPipelineRunsEnded` * `IntegrationTriggerRunsEnded` * `SQLStreamingBackloggedInputEventSources` * `SQLStreamingConversionErrors` * `SQLStreamingDeserializationError` * `SQLStreamingEarlyInputEvents` * `SQLStreamingInputEventBytes` * `SQLStreamingInputEvents` * `SQLStreamingInputEventsSourcesPerSecond` * `SQLStreamingLateInputEvents` * `SQLStreamingOutOfOrderEvents` * `SQLStreamingOutputEvents` * `SQLStreamingOutputWatermarkDelaySeconds` * `SQLStreamingResourceUtilization` * `SQLStreamingRuntimeErrors` ## microsoft.synapse/workspaces/bigdatapools * `BigDataPoolAllocatedCores` * `BigDataPoolAllocatedMemory` * `BigDataPoolApplicationsActive` * `BigDataPoolApplicationsEnded` ## microsoft.synapse/workspaces/scopepools * `ScopePoolJobPNMetric` * `ScopePoolJobQueuedDurationMetric` * `ScopePoolJobRunningDurationMetric` ## microsoft.synapse/workspaces/sqlpools * `ActiveQueries` * `AdaptiveCacheHitPercent` * `AdaptiveCacheUsedPercent` * `CPUPercent` * `Connections` * `ConnectionsBlockedByFirewall` * `DWULimit` * `DWUUsed` * `DWUUsedPercent` * `LocalTempDBUsedPercent` * `MemoryUsedPercent` * `QueuedQueries` * `WLGActiveQueries` * `WLGActiveQueriesTimeouts` * `WLGAllocationByEffectiveCapResourcePercent` * `WLGAllocationBySystemPercent` * `WLGEffectiveCapResourcePercent` * `WLGEffectiveMinResourcePercent` * `WLGQueuedQueries` ## microsoft.web/hostingenvironments * `CpuPercentage` * `DiskQueueLength` * `MemoryPercentage` * `TotalFrontEnds` ## microsoft.web/hostingenvironments/multirolepools * `ActiveRequests` * `AverageResponseTime` * `BytesReceived` * `BytesSent` * `CpuPercentage` * `DiskQueueLength` * `Http101` * `Http2xx` * `Http3xx` * `Http401` * `Http403` * `Http404` * `Http406` * `Http4xx` * `Http5xx` * `HttpQueueLength` * `HttpResponseTime` * `LargeAppServicePlanInstances` * `MediumAppServicePlanInstances` * `MemoryPercentage` * `Requests` * `SmallAppServicePlanInstances` * `TotalFrontEnds` ## microsoft.web/hostingenvironments/workerpools * `CpuPercentage` * `MemoryPercentage` * `WorkersAvailable` * `WorkersTotal` * `WorkersUsed` ## microsoft.web/serverfarms * `BytesReceived` * `BytesSent` * `CpuPercentage` * `DiskQueueLength` * `HttpQueueLength` * `MemoryPercentage` * `SocketInboundAll` * `SocketLoopback` * `SocketOutboundAll` * `SocketOutboundEstablished` * `SocketOutboundTimeWait` * `TcpCloseWait` * `TcpClosing` * `TcpEstablished` * `TcpFinWait1` * `TcpFinWait2` * `TcpLastAck` * `TcpSynReceived` * `TcpSynSent` * `TcpTimeWait` ## microsoft.web/sites * `AlwaysReadyFunctionExecutionCount` * `AlwaysReadyFunctionExecutionUnits` * `AlwaysReadyUnits` * `AppConnections` * `AverageMemoryWorkingSet` * `AverageResponseTime` * `BytesReceived` * `BytesSent` * `CpuPercentage` * `CpuTime` * `CurrentAssemblies` * `FileSystemUsage` * `FunctionExecutionCount` * `FunctionExecutionUnits` * `Gen0Collections` * `Gen1Collections` * `Gen2Collections` * `Handles` * `HealthCheckStatus` * `Http101` * `Http2xx` * `Http3xx` * `Http401` * `Http403` * `Http404` * `Http406` * `Http4xx` * `Http5xx` * `HttpResponseTime` * `InstanceCount` * `IoOtherBytesPerSecond` * `IoOtherOperationsPerSecond` * `IoReadBytesPerSecond` * `IoReadOperationsPerSecond` * `IoWriteBytesPerSecond` * `IoWriteOperationsPerSecond` * `MemoryWorkingSet` * `OnDemandFunctionExecutionCount` * `OnDemandFunctionExecutionUnits` * `PrivateBytes` * `Requests` * `RequestsInApplicationQueue` * `Threads` * `TotalAppDomains` * `TotalAppDomainsUnloaded` * `WorkflowActionsCompleted` * `WorkflowActionsFailureRate` * `WorkflowAppJobPullRate` * `WorkflowJobExecutionDelay` * `WorkflowJobExecutionDuration` * `WorkflowRunsCompleted` * `WorkflowRunsDispatched` * `WorkflowRunsFailureRate` * `WorkflowRunsStarted` * `WorkflowTriggersCompleted` * `WorkflowTriggersFailureRate` ## microsoft.web/sites/slots * `AppConnections` * `AverageMemoryWorkingSet` * `AverageResponseTime` * `BytesReceived` * `BytesSent` * `CpuTime` * `CurrentAssemblies` * `FileSystemUsage` * `FunctionExecutionCount` * `FunctionExecutionUnits` * `Gen0Collections` * `Gen1Collections` * `Gen2Collections` * `Handles` * `HealthCheckStatus` * `Http101` * `Http2xx` * `Http3xx` * `Http401` * `Http403` * `Http404` * `Http406` * `Http4xx` * `Http5xx` * `HttpResponseTime` * `IoOtherBytesPerSecond` * `IoOtherOperationsPerSecond` * `IoReadBytesPerSecond` * `IoReadOperationsPerSecond` * `IoWriteBytesPerSecond` * `IoWriteOperationsPerSecond` * `MemoryWorkingSet` * `PrivateBytes` * `Requests` * `RequestsInApplicationQueue` * `Threads` * `TotalAppDomains` * `TotalAppDomainsUnloaded` # Chronosphere Collector Source: https://docs.chronosphere.io/ingest/metrics-traces/collector Learn how the Chronosphere Collector scrapes Prometheus endpoints, receives traces, and sends telemetry data to Observability Platform. The Chronosphere Collector is a standalone app whose main function is to discover and scrape a Prometheus server that's exposed by your app. The Collector then sends any discovered data to the Chronosphere Observability Platform service through a connection secured by a revocable API token. If the Collector can't scrape certain jobs or components, you can manually push metrics to it. For more information, refer to [Service discovery](/ingest/metrics-traces/collector/discover). In addition to ingesting metrics, the Collector can: * Filter, rename, and drop Prometheus metrics that pass through it by using [Prometheus's relabeling rules](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config), based on specific label-value pairs. * Add [global labels](/ingest/metrics-traces/collector/configure/optimizations#add-global-labels) to all Prometheus metrics the Collector ingests. For example, you can append all Prometheus metrics passing through a particular Collector with the label `service="col1"`. * [Receive traces](/overview/types/traces) from your services, such as OpenTelemetry spans and other popular open source formats like Jaeger and Zipkin. Chronosphere supports Chronosphere Collector versions for a year from release. You can find a full list of versions and release dates in the Collector release notes located in the Chronosphere Observability Platform. To view the release notes, in the navigation menu select ** More information > Release notes**, and then click the **Collector** tab. ## Get started The Collector supports IPv4, IPv6, or fully qualified domain names for a single endpoint. To get started with the Collector: 1. [Review the architecture](#architecture) to determine how you want to install the Collector. 2. [Install the Collector](/ingest/metrics-traces/collector/install), either to retrieve metric data or receive trace data. 3. [Verify and monitor the Collector](/ingest/metrics-traces/collector/monitor) to ensure your data is streaming in to Observability Platform. 4. [Configure service discovery](/ingest/metrics-traces/collector/discover) to discover other apps to scrape metrics from. 5. [Enable recommended optimizations](/ingest/metrics-traces/collector/configure/optimizations#chronosphere-collector-optimizations) and modify default settings. 6. Optional: [Ingest additional metrics](/ingest/metrics-traces/collector/addl-metrics) such as DogStatsD, Graphite, Prometheus, or Pushgateway. If you're using Kubernetes, you can also [modify the scrape configuration](/ingest/metrics-traces/collector/discover/scrape-configuration) to override scrape defaults, change the annotation prefix, and define the `jobs` configuration. ## Architecture You can install the Collector with Kubernetes as a DaemonSet, Deployment, or as a sidecar. Each of these models have advantages and disadvantages, with different architectures. ### DaemonSet Installing the Collector as a Kubernetes DaemonSet is a pull-based model, where the Collector scrapes endpoints using either annotations or `ServiceMonitors`. With this method, you deploy a Collector on every node in your cluster. This method is the recommended way to install the Collector to receive metrics. ```mermaid actions={false} theme={null} graph TD accTitle: Chronosphere Collector DaemonSet architecture diagram accDescr: Architecture diagram that shows Collectors installed in each node of your Cluster, which then pass their data to the Observability Platform tenant. subgraph Kubernetes Cluster subgraph n1 [Node] a(( Pod   App )):::app b((Pod Storage)):::app c((Pod Chronosphere Collector)):::coll end subgraph n2 [Node] d(( Pod   App )):::app e((Pod Storage)):::app f((Pod Chronosphere Collector)):::coll end end c --> t(Observability Platform Tenant) f --> t classDef coll fill:#2fbf71,stroke:#495057,stroke-width:2px,color:#212529 classDef app stroke:#495057,stroke-width:2px style t fill:#60e684,stroke:495057,stroke-width:2px,color:#212529,stroke-dasharray: 5 5 ``` #### Advantages Because the Collector resides on the same Kubernetes node as your app, latency is low. As your cluster grows, deploy a Collector on each new node to scale your cluster horizontally. Installing one Collector per node also means that you're less likely to overload any single Collector. This method allows for phased rollouts of new Collector versions and configuration changes to help avoid losing scrape coverage. Also, if you lose a node, you're only losing metrics for that node and not your entire cluster. #### Disadvantages Some environments, such as AWS Fargate, restrict you from deploying an app as a DaemonSet. Kubernetes applies resource requirements globally across your cluster, which means that DaemonSets can evict other applications from the node depending on prioritization. Also, DaemonSets don't deal well with noisy scrape endpoints, such as `kube-state-metrics`. ### Deployment Installing the Collector as a Kubernetes Deployment works best for push-based models, where your app pushes data to a Kubernetes Service, which the Collector receives data from. This method is the recommended way to install the Collector to receive distributed trace data. To install the Collector in a Deployment where it pulls metrics data from scrape targets, Chronosphere recommends using Prometheus Service Discovery. If you don't need Pod metadata, you can install the Collector in a Deployment using ServiceMonitors. ```mermaid actions={false} theme={null} graph TD accTitle: Chronosphere Collector Kubernetes Deployment architecture diagram accDescr: Architecture diagram that shows the flow of data that is pushed to a Kubernetes Service, from which Collectors receive data and then pass to the Observability Platform tenant. subgraph Kubernetes Cluster subgraph Node a((Pod)):::app b((Pod)):::app c((Pod)):::app d((Pod)):::app e((Pod)):::app end a --> k((Kubernetes Service)):::app b --> k c --> k d --> k e --> k k --> a1((Chronosphere Collector)):::coll k --> a2((Chronosphere Collector)):::coll k --> a3((Chronosphere Collector)):::coll end a1 --> t(Observability Platform Tenant) a2 --> t a3 --> t classDef coll fill:#2fbf71,stroke:#495057,stroke-width:2px,color:#212529 classDef app stroke:#495057,stroke-width:2px style t fill:#60e684,stroke:495057,stroke-width:2px,color:#212529,stroke-dasharray: 5 5 ``` #### Advantages Deployments work well for monitoring high cardinality endpoints. You can choose to either configure multiple Collectors in your cluster or deploy a singular Collector. You can also select the number of Collectors to run in your cluster. As your cluster grows, add more Pods to scale your Deployment horizontally. #### Disadvantages If you deploy a singular Collector that's undersized for your workload, the Collector might experience delays in scrape latency or encounter out of memory situations. A Deployment can also be problematic when using staleness markers, and you can't customize the push configuration of the Collector per app. ### Sidecar A sidecar is a dedicated resource for your app that's ideal for handling high volume data, such an app that generates copious time series data. To run the Collector as a sidecar, you can install an individual Collector in each Pod of your cluster. Because the Collector Container and app Container run on the same Pod, you can optimize resources for the Pod. ```mermaid actions={false} theme={null} graph TD accTitle: Chronosphere Collector sidecar architecture diagram accDescr: Architecture diagram that shows running the Collector in a sidecar, with each Pod containing its own Collector, which then send information to the Observability Platform tenant. subgraph Kubernetes Cluster subgraph Node subgraph pod1 [Pod] a((App Container)):::app --> b((Chronosphere Collector)):::coll end subgraph pod2 [Pod] c((App Container)):::app --> d((Chronosphere Collector)):::coll end subgraph pod3 [Pod] e((App Container)):::app --> f((Chronosphere Collector)):::coll end end end b --> g(Observability Platform Tenant) d --> g f --> g classDef coll fill:#2fbf71,stroke:#495057,stroke-width:2px,color:#212529 classDef app stroke:#495057,stroke-width:2px style g fill:#60e684,stroke:495057,stroke-width:2px,color:#212529,stroke-dasharray: 5 5 ``` #### Advantages If a Collector goes offline, you lose only the metrics for a single Pod. You have more control over resource usage for high cardinality apps and can customize the Collector per app. #### Disadvantages Each Pod requires a sidecar Container, which requires additional Kubernetes resources for your cluster. Sidecars require you to maintain the app manifest to include the Collector as a sidecar, which increases overhead. Because the Collector exists on the same Pod as your app Container, the Collector can impact the entire Pod if it experiences problems. # Ingest additional metrics Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/addl-metrics Ingest Carbon, StatsD, SignalFx, Wavefront, and other metric formats with the Chronosphere Collector, and enable client-side aggregations. Scraping metric endpoints is the preferred method of ingesting metrics to the Chronosphere Collector, but you can also ingest the following metric formats: * [Carbon](/ingest/metrics-traces/collector/addl-metrics/carbon) * [DogStatsD](/ingest/metrics-traces/collector/addl-metrics/dogstatsd) * [Prometheus and OpenMetrics](/ingest/metrics-traces/collector/addl-metrics/prom-openmetrics) * [Pushgateway](/ingest/metrics-traces/collector/addl-metrics/prom-pushgateway) * [SignalFx](/ingest/metrics-traces/collector/addl-metrics/signalfx) * [StatsD](/ingest/metrics-traces/collector/addl-metrics/statsd) * [Tagged StatsD](/ingest/metrics-traces/collector/addl-metrics/tagged-statsd) * [Wavefront](/ingest/metrics-traces/collector/addl-metrics/wavefront) ## Enable aggregations You can enable client-side aggregations for [Carbon](/ingest/metrics-traces/collector/addl-metrics/carbon), [StatsD](/ingest/metrics-traces/collector/addl-metrics/statsd), and [Datadog DogStatsD](/ingest/metrics-traces/collector/addl-metrics/dogstatsd) metrics. These aggregations reduce network egress and processed writes to Chronosphere Observability Platform. When the Collector ingests a data sample, it aggregates the data based on the metric type and sends a single data point to Observability Platform that represents the sample for the defined period. The Collector aggregates on the following metric types: * Counters, which are rolled up through addition. * Gauges, which are selected by the last value written. * Timers, which use sketches of the underlying sample to build quantiles. Aggregations are disabled by default. To enable aggregations, define the `push.PROTOCOL.aggregation` YAML collection in your Collector manifest, where *`PROTOCOL`* can be `carbon`, `dogstatsd`, or `statsd`. For example, the following YAML collection enables aggregations for StatsD with defined values for each of the supported metric types: ```yaml theme={null} push: statsd: aggregation: counters: interval: "10s" gauges: interval: "10s" timers: interval: "10s" inactiveExpireAfter: "2m" ``` * `interval`: Defines the length of time before writing to Observability Platform. A longer `interval` duration provides more effective caching, but also increases the length of delays before writing. * `inactiveExpireAt`: Determines the length of time before Chronosphere evicts unused keys from the local cache. Default: `2m`. # Carbon ingestion Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/addl-metrics/carbon Configure the Chronosphere Collector to ingest Carbon metrics from Graphite clients as pre-aggregated gauge data. Carbon is an approach to sending Graphite data to the Collector. With this method, Chronosphere Observability Platform treats any Carbon data as pre-aggregated before it arrives at the Collector. Downsampling rules apply only for long-term retention with all metrics assumed to be gauges, while the short-term retention data is persisted as-is. Only use this when you are certain about the resolution of the data sent, with any change in resolution causing Observability Platform to persist more metrics. In general, Chronosphere doesn't recommend this approach of sending Graphite metrics, as the resolution and control of metrics ingestion is defined by the Carbon clients sending the data, and not the Collector or backend. You can enable [client-side aggregations](/ingest/metrics-traces/collector/addl-metrics#enable-aggregations) for Carbon. These aggregations reduce network egress and processed writes to Observability Platform. To ingest Carbon metrics with the Collector, add the following to the configuration file under the `push` YAML collection: ```yaml theme={null} push: carbon: enabled: true listenAddress: 0.0.0.0:3032 listenProtocol: udp ``` You can configure the following options in the Collector manifest for Carbon. * `listenAddress`: The address the server listens on. Point clients sending Carbon metrics to this address and port. Default: `0.0.0.0:3032`. When changing the default port, also update the container ports in the DaemonSet manifest to match. * `listenProtocol`: The protocol the server listens on. The Chronosphere Collector supports `udp`, `tcp,` or `tcp/udp` (both protocols on the same port). Default: `udp`. The Carbon server runs as a UDP server at the provided listen address. # DogStatsD ingestion Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/addl-metrics/dogstatsd Processes and configuration for ingesting DogStatsD metrics into Chronosphere Observability Platform. When accepting DogStatsD metrics over the User Datagram Protocol (UDP), Observability Platform requires you to run the Chronosphere Collector as a Kubernetes [DaemonSet](/ingest/metrics-traces/collector#deployment) or as a [sidecar](/ingest/metrics-traces/collector#sidecar). All ingested DogStatsD metrics are subject to sanitization rules that follow the [Datadog best practice guidelines](https://docs.datadoghq.com/developers/guide/what-best-practices-are-recommended-for-naming-metrics-and-tags/). For example, Chronosphere Observability Platform converts all DogStatsD labels to lowercase. For information about converting Datadog metrics and dashboards to Observability Platform, see the [Datadog overview](/ingest/metrics-traces/collector/mappings/datadog). The Collector supports DogStatsD protocols v1.0 through v1.6, and aggregates timer and distribution metrics as delta exponential histograms. ## Deploy the Chronosphere Collector Datadog metrics libraries primarily send metrics using the UDP protocol, which sends packets without reliability mechanisms, such as handshakes, retries, or acknowledgments. Using UDP to send data over the network increases the risk of losing metric data due to lost packets and other network issues. Because local UDP communication is generally more stable, keeping communication on the same Kubernetes node reduces this risk. ### Install Collector as a Kubernetes DaemonSet To install the Chronosphere Collector, follow the steps described in [Kubernetes Collector installation to retrieve metrics](/ingest/metrics-traces/collector/install/kubernetes). The manifest defines a ClusterRole with the permissions required to access the local Kubelet API and mounts a volume to access the `cgroups` directory on the node. ## Configure Chronosphere Collector for DogStatsD metrics 1. Use the configuration from the [annotated manifest](https://storage.googleapis.com/chronosphere-release/latest/kubernetes/chronocollector.yaml) in the [Kubernetes Collector installation documentation](/ingest/metrics-traces/collector/install/kubernetes) as a starting point. 2. Add `hostPort` to the DaemonSet `containters.ports` YAML collection to enable network communication on the host network. ```yaml theme={null} spec: template: spec: containers: ports: # ... - containerPort: 8125 hostPort: 8125 # Add hostPort name: dogstatsd protocol: UDP ``` 3. Enable DogStatsD mode under the `push.dogstatsd.enabled` YAML collection in the Collector manifest. ```yaml theme={null} push: dogstatsd: enabled: true ``` 4. Configure additional [DogStatsD configuration options](#dogstatsd-configuration-options) under the `push.dogstatsd` YAML collection as necessary. ## DogStatsD configuration options You can configure the following options in the Collector manifest for DogStatsD. Exception are denoted as "not configurable," which appear in code configuration output but can't be overridden by changing their values. In addition to augmenting labels with metadata from Kubernetes Pods, Collector v0.115.0 and later also supports label augmentation derived from the Kubernetes Node on which the Collector is running. Node metadata is fetched only once, at startup. * `listenAddress`: The address and port on which the server listens for connections. Point DogStatsD clients sending metrics to this address and port. Default: `0.0.0.0:8125`. When changing the default port, also update the container ports in the DaemonSet manifest to match. * `listenProtocol`: The protocol the server listens on. The Chronosphere Collector supports `udp`, `tcp,` or `tcp/udp` (both protocols on the same port). Default: `udp`. ```yaml theme={null} push: dogstatsd: enabled: true listenAddress: 0.0.0.0:8125 listenProtocol: udp ``` * `aggregation`: A YAML collection that defines how the Collector ingests a data sample. When Chronosphere Collector ingests a data sample, it aggregates the data based on the metric type and sends a single data point to Observability Platform that represents the sample for the defined `interval`. These aggregations reduce network egress and processed writes to Observability Platform. Chronosphere Collector aggregates data points based on the metric type as follows: * `counters` (not configurable): Sums all data point values for a time series and sends the `SUM` as a [`DELTA COUNTER`](/control/shaping/shape-metrics/types#delta-counter). * `gauges` (not configurable): Selects the `LAST` value for a time series in the interval. * `timers`: Aggregates timer values into a [`DELTA EXPONENTIAL HISTOGRAM`](/control/shaping/shape-metrics/types#delta-exponential-histograms), by default, including DogStatsD histograms and distributions. * `aggregationType`: Defines how the Collector aggregates timers. Default: `histogram`. The `tdigest` option is deprecated. Use `histogram` instead. * `interval`: Defines the length of time before writing to Observability Platform. Set an interval value that aligns with your licensed persistent writes and cardinality ratio. A more frequent aggregation interval increases persisted writes. * `inactiveExpireAfter`: Determines the amount of time before Chronosphere Collector evicts unused aggregation keys from the local cache. Default: `2m`. * `labels`: Adds labels to all DogStatsD metrics pushed. Define each label as a key-value pair. For example, the default configuration adds the labels `env` and `k8s_cluster_name` using environment variables for the values. * `prefix`: Adds a prefix to all DogStatsD metrics pushed. For example, setting this value to `qa` prefixes all metric names with `qa` followed by \`.\`, the dot separator. A metric like `build_info.ip` becomes `qa.build_info.ip` and is normalized to `qa_build_info_ip`. * `kubernetesMetadataAugmentation:` Defines the configuration and addition of Kubernetes metadata, labels, and annotations to be added to the time series. The Chronosphere Collector doesn't support using node or namespace metadata as the source for label augmentation. * `enabled`: A Boolean value that toggles augmentation. Default: `false`. The Chronosphere Collector retrieves metadata from the local Kubelet API and doesn't support other sources. * `kubelet`: * `kubeletNodeIP`: Defines the public IP address assigned to Kubelet. Defaults to the value of the `KUBERNETES_NODE_IP` environment variable. * `kubeletNodePort`: Defines the port that Kubelet listens on for HTTP requests. Default: `10250`. * `kubeletNodePodsEndpoint`: Overrides the URL of Kubelet's `/pods` endpoint, which by default is constructed from `kubeletNodePort` and `kubeletNodeIP` as `https://${kubeletNodePort}:${kubeletNodeIP}/pods`. * `kubeletNodeTLSInsecureSkipVerify`: Determines whether the Chronosphere Collector skips verification of Kubelet's TLS certificate. Default: `true`. * `kubeletNodeBearerTokenFile`: Defines the path to the file containing Collector's service account token. Default: `/var/run/secrets/kubernetes.io/serviceaccount/token`. * `timeout`: Defines the amount of time the Chronosphere Collector waits for a response from the metadata source. Default: `1s`. * `refreshInterval`: Defines the interval between refreshes if a cache miss didn't trigger a cache update from Kubelet. Default: `10s`. * `metadataToAugment`: Adds time series labels based on Kubernetes metadata from the Pod that sent the metrics or the Node on which the Collector is running. Node metadata is fetched only once, at startup. Supported metadata keys are `name` and `namespace`. * `labelsToAugment`: Adds time series labels based on Kubernetes labels from the Pod that sent the metrics or the Node on which the Collector is running. Node metadata is fetched only once, at startup. Specify the source Kubernetes label by name, and provide a name for the label as it should appear on the resulting metric time series. For example, the default configuration uses the value from the Kubernetes Pod label `app` as the value for the `service_name` label for all metrics. * `annotationsToAugment`: Adds time series labels based on Kubernetes annotations from the Pod that sent the metrics or the Node on which the Collector is running. Specify the source Kubernetes annotation by name, and provide a name for the label as it should appear on the resulting metric time series. For example, the default configuration uses the Kubernetes Pod annotation `app.myorg.com/owner` value as the value for the `team_owner` label. * `extractLabelsFromAnnotations`: A collection that extracts label-value pairs from Kubernetes Pod annotation values serialized as JSON key-value maps to add labels to time series ingested by the DogStatsD protocol handler. Set this to a `pod` containing `ad.datadoghq.com/tags` and `ad.datadoghq.com/%%container_name%%.tags`. This supports [Datadog Agent Tag Autodiscovery](https://docs.datadoghq.com/containers/kubernetes/tag/?tab=datadogoperator#tag-autodiscovery). * `cgroupRefreshInterval`: Defines how often to refresh the inode to cgroup mapping, if a cache miss didn't trigger a lookup. Default: `5m`. * `disableCgroupInodeResolution`: If `true`, disables the resolution of inodes to cgroups and containers. Default: `false`. * `stripContainerID`: Removes the `container_id` or `c` label sent by the DogStatsD metrics client. Default: `true`. ### Example configuration The following example YAML collection lists all the configuration options for receiving DogStatsD metrics with Chronosphere Collector: ```yaml theme={null} push: dogstatsd: enabled: true listenAddress: 0.0.0.0:8125 listenProtocol: tcp/udp aggregation: timers: aggregationType: histogram interval: 60s inactiveExpireAfter: "2m" labels: env: ${ENV:""} k8s_cluster_name: ${KUBERNETES_CLUSTER_NAME:""} # prefix: "qa" kubernetesMetadataAugmentation: enabled: true kubelet: kubeletNodeIP: ${KUBERNETES_NODE_IP} kubeletNodePort: 10250 # kubeletNodePodsEndpoint: https://${KUBERNETES_NODE_IP}:10250/pods kubeletNodeTLSInsecureSkipVerify: true kubeletNodeBearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token refreshInterval: 10s timeout: 1s # Note: Node metadata is fetched only once, at startup. metadataToAugment: pod: # map of metadata_keyname: metric_label name: kube_pod_name namespace: kube_namespace node: # map of node_metadata_key: metric_label name: kube_node_name labelsToAugment: pod: # map of pod_label: metric_label app: service_name node: # map of node_label: metric_label app: kube_node_name annotationsToAugment: pod: # map of pod_annotation: metric_label "app.myorg.com/owner": owner_team node: # map of node_annotation: metric_label "app.myorg.com/owner": node_owner extractLabelsFromAnnotations: pod: - ad.datadoghq.com/tags - ad.datadoghq.com/%%container_name%%.tags cgroupRefreshInterval: 5m # disableCgroupInodeResolution: false stripContainerID: true ``` # Prometheus and OpenMetrics ingestion Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/addl-metrics/prom-openmetrics Configure the Chronosphere Collector to accept text-exposition payloads in either Prometheus or OpenMetrics format. You can configure the Collector to accept text-exposition payloads in either Prometheus or OpenMetrics formats. To enable the Prometheus and OpenMetrics ingestion for the Collector, add `importPrometheus` and `importOpenMetrics` under the `push` YAML collection in the Collector configuration: ```yaml theme={null} push: importPrometheus: enabled: true importOpenMetrics: enabled: true ``` Traffic is served from `/import/openmetrics` and `/import/prometheus` from the `listenAddress`, which defaults to `0.0.0.0:3030`. ```shell theme={null} # Importing an untyped metric in OpenMetrics format. curl -d $'test_metric_import 100\n# EOF' -X POST http://localhost:3030/import/openmetrics # Same import, but in Prometheus format. curl -d 'test_metric_import 100' -X POST http://localhost:3030/import/prometheus ``` Additional query parameters: * `extra_label`: pass in repeated, default labels for all time series, formatted as `name=value`. * `default_timestamp`: provide a default timestamp in Unix milliseconds for any time series that doesn't have one. Otherwise, the value defaults to `Now()` on the Chronosphere backend. The following command includes both of these additional query parameters: ```shell theme={null} curl -d 'test_metric_import 100' -X POST \ http://localhost:3030/import/prometheus?extra_label=svc%3Dauth&extra_label=team%3Dbar&default_timestamp=1669701600 ``` ## Export Prometheus assets You can use [Chronoctl](/tooling/chronoctl) to sync Prometheus rule definitions and Alertmanager configuration into Observability Platform over the Chronosphere API. You need Chronoctl installed and configured. Read the [Chronoctl documentation](/tooling/chronoctl) for more details. For example, to export rules from a Prometheus Operator cluster and sync them: 1. In your Prometheus installation, retrieve your Prometheus rules into a YAML file: ```shell theme={null} kubectl get prometheusrules -o yaml -A > prometheus_rules.yml ``` 2. Gather the Alertmanager configuration you want evaluated with those rules (for example, your existing `alertmanager.yml`). 3. Run `chronoctl sync prometheus`. You must pass both Prometheus rules and Alertmanager YAML (`-p` and `-a`): ```shell theme={null} chronoctl sync prometheus -p prometheus_rules.yml -a alertmanager.yml ``` Chronoctl prints a changelog of changes applied in Observability Platform. To preview changes without writing them, add `--dry-run`. To control how rules map to teams and collections, pass a Chronosphere mapping file with `-c`: ```shell theme={null} chronoctl sync prometheus -p prometheus_rules.yml -a alertmanager.yml -c chronoconfig.yml ``` For flags and output formats, run `chronoctl sync prometheus --help` or see the [Chronoctl documentation](/tooling/chronoctl). # Pushgateway ingestion Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/addl-metrics/prom-pushgateway Learn how to configure and use the Prometheus Pushgateway to collect metrics. Scraping metrics is preferred whenever possible, but you can use the Prometheus Pushgateway for ephemeral jobs such as batch, AWS Lambda, or Google Cloud Functions that the Collector couldn't scrape otherwise. The Collector provides an HTTP endpoint that's compatible with the [Prometheus Pushgateway](https://github.com/prometheus/pushgateway), with the following caveats: * The open source Prometheus Pushgateway is a cache for metrics. Because the Collector is a proxy, it forwards metric payloads to the Chronosphere backend instead of caching metrics. This distinction means: * HTTP `DELETE` requests aren't supported. * HTTP `PUT` and `POST` requests are semantically the same. * The Query API, or `GET` requests, aren't supported. * Neither the Prometheus Pushgateway nor the Collector support any aggregation. * The Collector only supports GZIP compressed payloads. Snappy-compressed payloads aren't supported. * Transport Layer Security (TLS) isn't supported. All traffic to the Collector is in plain text, but traffic to the Chronosphere backend is always encrypted. Adhere to the following best practices when using the Prometheus Pushgateway with the Collector: * Ensure that at least one of the labels are unique to the instance of the calling job so that time series don't collide. If the label IDs are sufficiently high in cardinality, you might want to aggregate the labels. * Push metrics periodically, ideally when a job shuts down. * Run Collectors as sidecars (where all traffic is over `localhost`), or as a formal Kubernetes service. To enable the Prometheus Pushgateway for the Collector, modify the Collector configuration file and add the following lines: ```yaml theme={null} # This will register an HTTP endpoint on the listenerAddress for the Collector. push: pushgateway: enabled: true ``` ## Examples The following examples show how to send traffic with the Prometheus Pushgateway to the Collector, assuming that the listen address is running on port `3030`. ### Using curl When running this `curl` command, replace `` with the IP address of your Collector. ```shell theme={null} curl --data-binary @- http://:3030/pushgateway/metrics/job/some_job/job_id/my_unique_id <` is the name of the Kubernetes pod where the Collector is running: ```shell theme={null} kubectl port-forward 3030 ``` You can then run the previous `curl` command and use `localhost` instead of the Collector IP address. ### Using the Prometheus Pushgateway client ```go theme={null} import ( "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/push" ) func main() { registry := prometheus.NewRegistry() // Push metrics on shutdown. defer func() { // The address can be replaced with a Kubernetes Service name, // however, the URL must end in /pushgateway. push.New("http://localhost:3030/pushgateway", "my_job"). Grouping("job_id", "my_unique_id").Gatherer(registry).Push() }() } ``` # SignalFx ingestion Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/addl-metrics/signalfx Configure the Chronosphere Collector to process SignalFx data points from apps instrumented with a SignalFx client library. You can configure the Collector to process SignalFx data points. The Collector serves SignalFx traffic at the `/signalfx/v2/datapoint` HTTP endpoint from the `listenAddress`, which defaults to `0.0.0.0:3030`. The `/v2/datapoint` suffix is the default expected path appended by SignalFx client libraries. This integration pattern assumes that you've instrumented your apps with a SignalFx client library, such as the [golib](https://github.com/signalfx/golib) library for Go. For equivalent functions between SignalFx and PromQL, see the [equivalent function list](/ingest/metrics-traces/collector/mappings/signalfx). ## Configure SignalFx ingestion 1. Download this [annotated manifest](https://storage.googleapis.com/chronosphere-release/latest/kubernetes/chronocollector.yaml) as a starting point. 2. Enable SignalFx mode under the `push.signalFX.enabled` YAML collection in the Collector manifest. ```yaml theme={null} push: signalFX: enabled: true ``` 3. Modify additional configuration options under the `push.signalFX` YAML collection. # Receive and aggregate metrics from StatsD Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/addl-metrics/statsd Processes and configuration for receiving StatsD metrics to Chronosphere Observability Platform. The Chronosphere Collector supports Graphite metrics and two methods of configuring Graphite-related ingestion: StatsD or [Carbon](/ingest/metrics-traces/collector/addl-metrics/carbon). To enable or make changes to Graphite ingestion, contact [Chronosphere Support](/support). StatsD sends unaggregated data to the Chronosphere Collector, which aggregates StatsD metrics at a configurable interval and sends them to Chronosphere Observability Platform. ## Deploy the Chronosphere Collector StatsD metrics libraries primarily send metrics using the User Datagram Protocol (UDP), which sends packets with no handshakes, retries, acknowledgments, or other reliability mechanisms. Using UDP to send data over the network increases the risk of losing metrics data due to lost packets or other network issues. Keeping communication on the same Kubernetes node reduces these issues because local UDP communication is generally more stable. ### Install Collector as a Kubernetes DaemonSet To install the Chronosphere Collector, follow the steps described in [Kubernetes Collector installation to retrieve metrics](/ingest/metrics-traces/collector/install/kubernetes). ## Configure Chronosphere Collector for StatsD metrics Perform the following steps to configure Chronosphere Collector for StatsD metrics. 1. Use the configuration in the [annotated manifest](https://storage.googleapis.com/chronosphere-release/latest/kubernetes/chronocollector.yaml) from [Kubernetes Collector installation to retrieve metrics](/ingest/metrics-traces/collector/install/kubernetes) as a starting point. 2. Enable StatsD mode using the `push.statsd.enabled` YAML collection in the Collector manifest: ```yaml theme={null} push: statsd: enabled: true ``` 3. Configure additional [StatsD configuration options](#statsd-configuration-options) under the `push.statsd` YAML collection as necessary. ## StatsD configuration options You can configure the following options in the Collector manifest for StatsD, with the exception of those denoted in the following list as not configurable, which appear in code configuration output but can't be overridden by changing their values. * `enabled`: A Boolean option that determines whether to enable StatsD mode under the `push.statsd.enabled` YAML collection in the Chronosphere Collector configuration. Default: `true`. * `listenAddress`: The address and port on which the server listens for connections. Point StatsD clients sending metrics to this address and port. Default: `0.0.0.0:3031`. * `listenProtocol`: Defines whether to use `udp` or `tcp` protocols. Default: `udp`. * `aggregation`: A YAML collection that defines how the Collector aggregates data samples. When Chronosphere Collector ingests a data sample, it aggregates the data based on the metric type and sends a single data point to Observability Platform that represents the sample for the defined `interval`. These aggregations reduce network egress and processed writes to Observability Platform. Chronosphere Collector aggregates data points based on the metric type as follows: * `counters` (not configurable): Sums all data point values for a time series and sends the `SUM` as a [`DELTA COUNTER`](/control/shaping/shape-metrics/types#delta-counter). * `gauges` (not configurable): Selects the `LAST` value for a time series in the interval. * `timers`: A YAML collection that defines the aggregation type of timers. By default, Chronosphere Collector doesn't aggregate timer values. Aggregation rules are required to aggregate timers at ingestion. * `aggregationType`: Defines how Chronosphere Collector aggregates timers. Default: `none`, which supports using Graphite to query metrics. Set this value to `histogram` to instead send a [`DELTA EXPONENTIAL HISTOGRAM`](/control/shaping/shape-metrics/types#delta-exponential-histograms) that can be queried only by using PromQL. * `interval`: Defines the amount of time before writing to Observability Platform. Use an interval value that aligns with your licensed persisted writes and cardinality ratio. A more frequent aggregation interval increases persisted writes. Default: `60s`. * `inactiveExpireAfter`: Determines the amount of time before Chronosphere Collector evicts unused aggregation keys from the local cache. Default: `2m`. ## Example configuration The following example YAML collection lists all of the configuration options for receiving StatsD metrics with Chronosphere Collector: ```yaml theme={null} push: statsd: enabled: true listenAddress: 0.0.0.0:3031 listenProtocol: udp aggregation: timers: # Valid values are "histogram" and "none" aggregationType: none interval: 60s inactiveExpireAfter: "2m" ``` # Wavefront Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/addl-metrics/wavefront Configure the Chronosphere Collector to accept metric payloads in Wavefront data format, including histogram distributions. You can configure the Collector to accept metric payloads in Wavefront data format, instrumented with a Wavefront client library, such as the [wavefront-sdk-go](https://github.com/wavefrontHQ/wavefront-sdk-go) library. ## Basic configuration and data format To enable Wavefront ingestion for the Collector, add the `wavefront` configuration block under the `push` key in the Collector configuration: ```yaml theme={null} push: wavefront: enabled: true ``` The metrics ingestion protocol used by Wavefront requires clients to submit their metrics to compatible backends using `HTTP POST`. The payload should be sent as text in the Wavefront data format. Each `HTTP POST` body can contain multiple metrics and distributions, separated by newlines. Learn more about the [Wavefront Data Format](https://docs.wavefront.com/wavefront_data_format.html) and [Sending Histogram Distributions](https://docs.wavefront.com/proxies_histograms.html#sending-histogram-distributions). The Collector Wavefront endpoint serves from the path `/wavefront/report` from the `listenAddress`, which defaults to `0.0.0.0:3030`. The Collector implementation for Wavefront can parse `collectDLabels` from the metric name and store them as tags for the time series. For example, a metric name like `k8s.service1[label1=value1,label2=value2]test` produces a metric name of `k8s_service1test`, with labels `label1=value1` and `label2=value2`. ## Advanced configuration options * `mode` The mode field is optional and its default value is `regular`. The field can take other values, such as `graphite` and `graphite_expanded`. This field controls metric name sanitizing and storage in Chronosphere. Similar to the DogStatsD configuration, Wavefront configuration also supports this field and exhibits similar behavior. ```yaml theme={null} push: wavefront: enabled: true mode: graphite ``` * `defaultHistogramBuckets` The histograms' default upper bounds for distribution-type metrics span from two milliseconds to 3,600 seconds. In most cases, the default settings are sufficient. If necessary, adjust these default values using the `defaultHistogramBuckets` key in the configuration, as demonstrated here: ```yaml theme={null} push: wavefront: enabled: true mode: regular defaultHistogramBuckets: - upper: 0.004 - upper: 0.006 - upper: 0.008 - upper: 0.01 - upper: 0.02 - upper: 0.04 - upper: 0.06 - upper: 0.08 - upper: 0.1 - upper: 0.2 - upper: 0.4 - upper: 0.6 - upper: 0.8 - upper: 1.0 - upper: 1.5 - upper: 2.0 - upper: 2.5 - upper: 3.0 - upper: 3.5 - upper: 4.0 - upper: 4.5 - upper: 5.0 - upper: 5.5 - upper: 6.0 - upper: 6.5 - upper: 7.0 - upper: 7.5 - upper: 8.0 - upper: 8.5 - upper: 9.0 - upper: 9.5 - upper: 10 - upper: 15 - upper: 20 - upper: 25 - upper: 30 - upper: 35 - upper: 40 - upper: 45 - upper: 50 - upper: 55 - upper: 60 - upper: 150 - upper: 300 - upper: 450 - upper: 600 - upper: 900 - upper: 1200 - upper: 1500 - upper: 1800 - upper: 2100 - upper: 2400 - upper: 2700 - upper: 3000 - upper: 3300 - upper: 3600 ``` # Configure your Chronosphere Collector Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/configure Define a Chronosphere Collector manifest that sets the installation method and scrape endpoints, then update it in your cluster. To install and use the Chronosphere Collector, you define a manifest that describes how the Collector operates. Manifests are static YAML files that include details such as the installation method for the Collector and what endpoints the Collector scrapes to ingest data. You can base your manifest on [this example](https://storage.googleapis.com/chronosphere-release/latest/kubernetes/chronocollector.yaml), which contains details about common configuration options when installing the Collector with Kubernetes to retrieve metric data. You can modify the default Collector manifest to: * [Apply recommended optimizations](/ingest/metrics-traces/collector/configure/optimizations). * [Monitor Kubernetes resources](/ingest/metrics-traces/collector/discover/monitor-kubernetes). ## Modify the Collector manifest If you modify the Collector configuration manifest, you must send the updated manifest to the cluster and then restart the Collector: 1. Apply the updated manifest to the cluster: ```shell theme={null} kubectl apply -f path/to/manifest.yml ``` 2. Restart the Collector to propagate the changes: **DaemonSet** ```shell theme={null} kubectl rollout restart daemonset/chronocollector ``` **Deployment** ```shell theme={null} kubectl rollout restart deployment/chronocollector ``` # Scrape Prometheus native histograms Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/configure/native-histograms Configure the Chronosphere Collector to scrape Prometheus native histograms, and control histogram cost and backward compatibility. To scrape Prometheus native histograms, you must configure the Collector to enable native histogram scraping. Observability Platform persists Prometheus native histograms as a [histogram metric type](/control/shaping/shape-metrics/types#histogram). You can define aggregation rules to control histogram costs. For more information, see [supported histogram aggregation operations](/control/shaping/shape-metrics/rules/rollup#histograms-aggregation-operations). Histogram persisted writes, persisted cardinality, and matched writes consume [Histogram Metric License](/administer/limits-licensing/concepts/capacity-licensing#histogram-metrics-license) capacity. ## Backward compatibility Prometheus clients don't support exposing native histograms in the text-based exposition format. To scrape native histograms, Chronosphere Collector negotiates the Protobuf format for scraping. The change to scrape the Protobuf format can cause problems for backward compatibility. Depending on the Prometheus client, the value for classic histogram `le` labels and summary quantile labels appear as an integer in text format (`le="50"`) and a decimal number (`le="50.0"`) in Protobuf format. Because these values are strings, queries that filter by the integer label value stop working when the `le` values are in a decimal number format. For more information, see the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/feature_flags/#native-histograms). The Chronosphere Collector provides backward compatibility by dual-scraping the text-based format and the Protobuf format. The Collector first scrapes the Protobuf exposition and keeps only native histogram metrics. The Collector then scrapes the text-based format for all other metrics to preserve the `le` value format in histogram and quantile time series. Dual-scraping ensures backward compatibility and adds only minor overhead. ## Migrate from classic Prometheus histograms to native histograms The Prometheus Java and Go clients support Prometheus native histograms. Because native histograms are an experimental feature, use the latest version of the Prometheus Java and Go clients for the best experience. ### Prometheus client configuration The Prometheus Java client maintains both [classic and native histogram representation](https://prometheus.github.io/client_java/api/io/prometheus/metrics/core/metrics/Histogram.html) by default. It exposes the classic histogram representation in the text-based format and the native histogram representation in the Protobuf format. The Go Prometheus client can also expose both classic and native histogram representations. However, unlike the Java client it requires you to opt-in to each bucket layout. To make the classic histogram available for scraping, users must explicitly define the [custom bucket layouts](https://github.com/prometheus/client_golang/blob/6e3f4b1091875216850a486b1c2eb0e5ea852f98/prometheus/histogram.go#L394). To make the native histogram representation available, users must set a value for [`NativeHistogramBucketFactor`](https://github.com/prometheus/client_golang/blob/6e3f4b1091875216850a486b1c2eb0e5ea852f98/prometheus/histogram.go#L399). The `Buckets` field configures classic histograms. To collect exponential histograms, use the `NativeHistogramBucketFactor` parameter. ### Chronosphere Collector configuration Start by configuring the Chronosphere Collector to collect both the classic and native histogram representations. After you've built up enough historical data and modified your queries to use native histograms, you can modify the Collector configuration to scrape only the Protobuf format. Collect both classic and native histogram representations for at least as long as your query time windows. For example, if you have service-level objective queries with a 30-day look-back period, collect both representations for at least 31 days before changing your settings to scrape only the Protobuf format. ### Recommended initial configuration Chronosphere recommends the following Collector configuration when initially transitioning from classic histograms to native histograms. The configuration sets up the Collector to: * Negotiate scraping both the Protobuf and text-based expositions. * Scrape native histograms from the Protobuf exposition when present. * Scrape all other metrics from the text-based exposition. * Send both the classic and native histogram representations to Observability Platform, when a histogram exists in both formats. ```yaml theme={null} scrape: nativeHistograms: enabled: true # Enables scraping native histograms. Defaults to false. defaults: scrapeClassicHistograms: true # Sends both the classic and native histogram # representation when both are present. Defaults to false. ``` ### Fully-migrated configuration After you've collected enough histogram data and have modified your queries to query the native histogram representation, you can configure the Collector to no longer send the classic histogram representation. The following configuration configures the Collector to: * Negotiate scraping both the Protobuf and text-based expositions. * Scrape all metrics from the Protobuf exposition when present. ```yaml theme={null} scrape: nativeHistograms: enabled: true # Enables scraping native histograms. Defaults to false. ``` # Chronosphere Collector optimizations Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/configure/optimizations Apply recommended and optional Chronosphere Collector optimizations, such as ingestion buffering, for Kubernetes and Prometheus. The following configurations provide different optimizations for the Chronosphere Collector. Although their use isn't required, some of these configurations are recommended, and all of them can optimize the Collector in different ways, depending on your needs. These configurations apply to running the Collector with either Kubernetes or Prometheus. If you modify a Collector manifest, you must [update it in the cluster and restart the Collector](/ingest/metrics-traces/collector/configure#modify-the-collector-manifest). ## Recommended optimizations The following configurations are recommended for use with the Collector. ### Enable ingestion buffering The Collector can retry a subset of metric upload failures (explicitly excludes rate-limited uploads and malformed metrics). To configure ingestion buffering, create a writable directory and pass it to the Collector to store this data. ```yaml theme={null} ingestionBuffering: retry: enabled: true directory: /path/to/WRITE_DIRECTORY # How long individual metric uploads should be retried before being considered permanently failed # Values greater than 90 seconds may lead to unexpected behavior and are not supported defaultTTLInSeconds: 90 # The Collector will use strictly less than this amount of disk space. maxBufferSizeMB: 1024 ``` Replace *`WRITE_DIRECTORY`* with the writable directory the Collector can use for ingestion buffering. With Kubernetes, you can use an [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume mount. You can disable ingestion buffering for individual types of metrics. For example, to disable ingestion buffer retries for Carbon metrics, add a `push.carbon` YAML collection to the Collector configuration file and define ingestion buffering for Carbon metrics: ```yaml theme={null} push: carbon: retry: disabled: true # You can also adjust the TTL for this type of metric here # ttlInSeconds: 30 ``` ### Configure connection pooling Connection pooling is enabled by default with a pool size of `1`. A single Collector instance is capable of high throughput. However, if the Collector sends metrics at more than 100 requests per second, increase the number of pooled backend connections to improve overall throughput in the client. If you enable self-scraping, you can submit the following query with Metrics Explorer to verify the connection pooling setting: ```text theme={null} sum by(instance) (rate(chronocollector_gateway_push_latency_count[1m])) > 100 ``` To configure connection pooling, add the following YAML collection and define the values appropriately: ```yaml theme={null} backend: connectionPooling: # Enables connection pooling. By default, this is enabled. enabled: true # The pool size is tunable with values from [1,8]. If not specified and pooling is enabled, then # the default size is 1. poolSize: 1 ``` ### Enable staleness markers When a scrape target disappears or doesn't return a sample for a time series that was present in a previous scrape, queries return the last value. After five minutes, queries return no value, which means queries might return out-of-date data. By enabling staleness markers, the Collector can hint to the database that a time series has gone stale, and exclude it from query results until it reappears. A staleness marker gets published when the target disappears or doesn't return a sample. Staleness markers are disabled by default in the Collector configuration. Staleness markers are a best effort optimization. If a Collector instance restarts on the last scrape before a sample isn't provided for a time series, a staleness marker isn't published. There is a memory cost to enabling staleness markers. The memory increase is dependent on the time series scraped and their labels. For example, if the Collector is scraping 500 time series per second, memory usage increases by about 10%. If it's scraping 8,000 time series per second, memory usage increases by about 100%. If the Collector has self-scraping enabled, submit the following query with Metrics Explorer to review the scraped time series: ```text theme={null} rate(chronocollector_scrape_sample[5m]) ``` To enable staleness markers, add the following YAML collection to your Collector configuration: ```yaml theme={null} scrape: enableStalenessMarker: true ``` ## Additional configurations The following configurations are available for your use, as needed. ### Modify the default compression algorithm The Collector uses [Zstandard](https://github.com/facebook/zstd) (`zstd`) as the default compression algorithm instead of `snappy`. The `zstd` algorithm can greatly reduce network egress costs, which can reduce the data flowing out of your network by up to 60% compared to `snappy`. On average, `zstd` requires about 15% more memory than `snappy`, but offers a compression ratio that's 2.5 times greater. By default, zstd compression concurrency is capped at 1, and all requests must synchronize access. This limits the memory overhead and CPU processing required for compression. This can also reduce throughput, although the reduction is limited. If your Collector encounters processing bottlenecks, you can increase the concurrency value: ```yaml theme={null} backend: zstd: concurrency: 1 ``` Similarly, you can tune the compression level. With a `level` setting, which supports a range of values `["fastest", "default", "better", "best"]` that provide increasing orders of compression. The Collector defaults to `default`, which corresponds to Level 3 zstd compression. `best` strives for the best compression regardless of CPU cost, and `better` typically increases the CPU cost by 2-3x. The 2.5x improvement was achieved with `level: default` compression and `concurrency: 1`. ```yaml theme={null} backend: zstd: concurrency: 1 level: "default" # fastest, default, better, and best are acceptable values. ``` The following graph shows the compression difference between using `zstd` instead of `snappy` as the default compression algorithm. Although `snappy` provides 10 times (10x) the amount of compression, `zstd` provides roughly twenty-five times (25x) compression. The compression savings realized in your environment greatly depends on the format of your data. For example, the Collector can achieve higher compression with Prometheus data, but each payload contains more data than StatsD. Graph showing the compression gains of roughly 2.5 more when using zstd over snappy as the default compression algorithm If this tradeoff doesn't work for your environment, you can modify the Collector configuration file to instead use `snappy`: ```yaml theme={null} backend: compressionFormat: "snappy" ``` ### Implement environment variables Environment variable expansion is a powerful concept when defining a Collector configuration. Expansions use the syntax `${ENVIRONMENT_VARIABLE:"VALUE"}`, which you can use anywhere to define per-environment configurations dynamically. For example, the Collector manifests provided in the [Kubernetes installation](/ingest/metrics-traces/collector/install/kubernetes) page include an environment variable named `KUBERNETES_CLUSTER_NAME` that refers to the Kubernetes namespace. You can define a value for this variable in your Collector manifest under the `spec.template.spec.containers.env` YAML collection: ```yaml theme={null} spec: template: spec: containers: env: - name: KUBERNETES_CLUSTER_NAME value: YOUR_CLUSTER_NAME ``` Replace *`YOUR_CLUSTER_NAME`* with the name of your Kubernetes cluster. Refer to the [Go `Expand` documentation](https://pkg.go.dev/go.uber.org/config#Expand) for more information about environment variable expansion. [Environment variable expansions](/ingest/metrics-traces/collector/configure/optimizations#implement-environment-variables) and [Prometheus relabel rule](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) regular expression capture group references can use the same syntax. For example, `${1}` is valid in both contexts. If your relabel configuration uses Prometheus relabel rule regular expression capture group references, and they are in the `${1}` format, escape the syntax by adding an extra `$` character to the expression such as `$${1}`. ### Define the `listenAddress` The `listenAddress` is the address that the Collector serves requests on. It supports the `/metrics` endpoint and the [import endpoints](/ingest/metrics-traces/collector/addl-metrics/prom-openmetrics#prometheus-and-openmetrics-ingestion) if enabled. You can also configure the `listenAddress` by using the environment variable `LISTEN_ADDRESS`. The default value is `0.0.0.0:3030`. ### Set the logging level You can control the information that the Collector emits by setting a logging level in the configuration file. To set a logging level, add the following YAML collection to your configuration: ```yaml theme={null} logging: level: ${LOGGING_LEVEL:LEVEL} ``` Replace *`LEVEL`* with one of the following values. Use the `info` logging level for general use. | Logging level | Description | | ------------- | ------------------------------------------------------------------------------------------ | | `info` | Provides general information about state changes, such as when adding a new scrape target. | | `debug` | Provides additional details about the scrape discovery process. | | `warn` | Returns information related to potential issues. | | `error` | Returns error information for debugging purposes. | | `panic` | Don't use this logging level. | #### Temporarily change the Collector logging level The Collector exposes an HTTP endpoint available at the `listenAddress` that temporarily changes the logging level of the Collector. The `/set_log_level` endpoint accepts a JSON body with parameters. The following request sets the logging level to `debug` for a duration of 90 seconds: ```shell theme={null} curl -X PUT http://localhost:3030/set_log_level -d '{"log_level": "debug", "duration": 90}' ``` * `log_level`: Required. Defines the logging level. * `duration`: Optional: Defines the duration to temporarily set the logging level for, in seconds. Default: `60`. If you send a new request before a previous request's duration has expired, the previous request is overridden with the latest request's parameters. ### Add global labels You can add global or default labels using: * [A static list](#labels-from-a-configuration-list) * [An external JSON file](#labels-from-an-external-file) * [Both a static list and an external file](#labels-from-both-a-configuration-list-and-an-external-file) If you define a global label with the same name as the label of an ingested metric, the Collector respects the label for the ingested metric and doesn't overwrite it. #### Labels from a configuration list If you're using either Kubernetes or Prometheus discovery, you can add default labels as key-value pairs under the `labels.defaults` YAML collection: ```yaml theme={null} labels: defaults: my_global_label: ${MY_VALUE:""} my_second_global_label: ${MY_SECOND_VALUE:""} ``` If you're using Kubernetes, you can append a value to each metric sent to Chronosphere Observability Platform by adding the `KUBERNETES_CLUSTER_NAME` environment variable as a default label under the `labels.defaults.tenant_k8s_cluster` YAML collection: ```yaml theme={null} labels: defaults: tenant_k8s_cluster: ${KUBERNETES_CLUSTER_NAME:""} ``` Refer to the [Kubernetes documentation](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) for more information about pod fields you can expose to the Collector within manifest. For Prometheus discovery, you can add labels to your job configuration using the `labels` YAML collection. For example, the following configuration adds `rack` and `host` to every metric: ```yaml theme={null} static_configs: - targets: ['0.0.0.0:9100'] labels: host: 'foo' rack: 'bar' ``` #### Labels from an external file You can define labels in an external JSON file in the `labels.file` YAML collection: ```yaml theme={null} labels: file: "labels.json" ``` You then add key-value pairs in the `labels.json` JSON file: ```json theme={null} { "default_label_1": "default_val_1", "default_label_2": "default_val_2" } ``` #### Labels from both a configuration list and an external file If you specify labels in both the configuration and an external file, the Collector uses the combined list of default labels, if there are no duplicated keys defined with both methods. If you define a label key both in the static list in configuration and the external JSON file, the Collector reports an error and fails to start. To specify default labels in both input sources: * Add key-value pairs under the `label.defaults` YAML collection. * Specify an external JSON file in the `labels.file` YAML collection. ```yaml theme={null} labels: defaults: default_label_1: "default_val_1" default_label_2: "default_val_2" file: "labels.json" ``` You then add key-value pairs in the `labels.json` JSON file: ```json theme={null} { "default_label_3": "default_val_3", "default_label_4": "default_val_4" } ``` In this example, the Collector uses all four default labels defined. ### Configure runtime memory limits The Collector sets a runtime memory limit of 85% of the container (process cgroup) memory quota under Linux, allowing automatic tuning outside of Kubernetes installations. You can customize these limits by configuring settings in the `performance` section of the Collector configuration. ```yaml theme={null} performance: # enforceSoftMemoryLimit enables automatic turning of GOMEMLIMIT from environment # (Linux cgroups) limits. Enabled by default. enforceSoftMemoryLimit: true # reservedMemoryPercent controls how much memory to set aside for non-heap use, # if memory quota was autodetected from cgroup limits. # Go runtime memory limit will be set to $quota - ($quota * reservedMemoryPercent / 100). # For more information, see https://go.dev/doc/gc-guide#Memory_limit reservedMemoryPercent: 15 ``` ### Configure metrics batching > Requires [Chronosphere Collector](/ingest/metrics-traces/collector) version 0.114.0 or later. The Collector batches metrics across requests to improve efficiency when sending metrics to Observability Platform. For example, the Collector sends a batch of metrics scraped from multiple scrape jobs in a single request to Chronosphere's metrics ingestion endpoint, rather than creating separate requests to send metrics from each scrape job. The Collector similarly batches metrics from push protocols, such as Pushgateway and DogStatsD. The Collector batches metrics on a per-protocol basis, with separate batches for Prometheus scraped metrics, Pushgateway, DogStatsD, and other protocols. The `requestBatching` settings, configured in the `backend` YAML collection, apply globally to each independent protocol queue. **Don't change the default settings** unless advised to do so by Chronosphere Support. The following example documents these settings but intentionally comments them out. ```yaml theme={null} # backend: # requestBatching: # disabled: false # # maxConcurrentRequests limits the max number of in-flight requests. # maxConcurrentRequests: 50 # # maxBufferSize is a limit in bytes for in-flight data payloads. Requests are # # rejected if an amount of bytes larger than this value is already in flight, # # similar to `maxConcurrentRequests`. # maxBufferSize: 1073741824 # # batchTimeout sets the maximum delay for requests if the minimum batch threshold # # hasn't been reached. # batchTimeout: 5s # # maxRequestSize is the threshold in bytes that triggers a backend request. # maxRequestSize: 131072 ``` # Debug endpoints Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/debugging How to debug Chronosphere Collector issues by using available HTTP endpoints. The Chronosphere Collector exposes HTTP endpoints to return information about the Collector that can help with debugging issues. The Collector enables these endpoints by default. To disable them, set [`debug.disabled`](#debug-configuration) to `true` in the Collector manifest. In Collector version 0.118.0 and later, debug endpoints are bound to loopback or localhost by default. For more information about the settings to override this default behavior, see [Debug configuration](#debug-configuration). ## Access the debug endpoints All endpoints except `/set_log_level` are clickable links on the `/debug` HTML page, and are also directly accessible with a `GET` request. 1. In the Collector manifest, enable Prometheus discovery and configure at least one scrape configuration: You must complete this step to access the debug endpoints, regardless of whether you're using the Collector to scrape Prometheus endpoints for metrics. ```yaml theme={null} prometheus: enabled: true scrape_configs: - job_name: 'access_debug' scrape_interval: 30s scrape_timeout: 15s static_configs: - targets: ['192.168.192:9100'] ``` 2. To establish a connection with the Collector, use a method compatible with your Collector installation method. For Kubernetes installations, set up port forwarding to access the debug endpoint: ```shell theme={null} kubectl port-forward chronocollector-POD_NAME LISTEN_PORT ``` Replace the following: * *`POD_NAME`*: The name of the Kubernetes pod where your Collector instance is running. * *`LISTEN_PORT`*: The port where the Collector serves debug requests. The value of [`debug.listenAddress`](#debug-configuration) defines this port. If no value is set, the default port is `3030`. For standalone installations, use one of the following methods: * (Recommended) Use SSH to directly access your host. * Configure the Collector to listen to a specific address by setting a value for [`debug.listenAddress`](#debug-configuration). * Configure the Collector to set [`debug.exposeDebugPagesOnAllInterfaces`](/ingest/metrics-traces/collector/debugging#debug-configuration) to `true`. Use with caution, as enabling this setting can publicly expose debug endpoints. 3. Open your browser and navigate to `HOSTPORT/debug`. Replace *`HOSTPORT`* with the relevant value based on the method and configuration settings you used in step 2: Replace *`HOSTPORT`* with the value `localhost:3030`. * If you used SSH to directly access your host, replace *`HOSTPORT`* with the value `localhost:3030`. * If you set a value for `debug.listenAddress`, replace *`HOSTPORT`* with the same value you set for `debug.listenAddress`. * If you set `debug.exposeDebugPagesOnAllInterfaces` to `true`, replace *`HOSTPORT`* with any valid address that your server listens on. To make a `GET` request to any debug endpoint, query `http://HOSTPORT/debug/`, replacing *`HOSTPORT`* with the relevant value. 4. Click one of the [available endpoints](#available-debug-endpoints) from the list. ## Available debug endpoints * `/debug` Displays an HTML page that lists all debug endpoints, each of which is accessible by clicking an endpoint from the list. This page also displays a table of active scrape targets, which includes two columns: * **Target Info:** Lists configuration details for each scrape target. * **Last Scrape Result:** For each scrape outcome, lists information about that outcome. If available, a decoded body sample shows a portion of the most recent scrape output. * `/debug/bundle` Downloads a `tar.gz` file that contains the output of all debug endpoints. The file format is `node-name--date--time.tar.gz`, where `node-name` is the name of the node the Collector runs in. * `/debug/collector_runtime_memory_stats` Returns runtime memory metrics for the Collector. * `/debug/config_dump` Returns a JSON representation of the Collector configuration settings defined in the `raw_config`, combined with the values for those settings calculated at runtime. Responses from this endpoint might contain sensitive credentials. In Collector version 0.118.0 and later, this endpoint is disabled by default. To enable it, set `debug.enableRawConfig` to true. * `/debug/endpoint_slices` Returns a JSON list of all Kubernetes EndpointSlices the Collector detects. * `/debug/job_configs` Returns a JSON list of all computed Kubernetes jobs created from the Collector configuration. * `/debug/pods` Returns a JSON list of all Kubernetes pods the Collector detects. * `/debug/pprof` Serves a protocol buffer of the Collector in [pprof](https://github.com/google/pprof) format. * `/debug/raw_config` Returns the raw Collector configuration file as-is, without runtime-computed values. Responses from this endpoint might contain sensitive credentials. In Collector version 0.118.0 and later, this endpoint is disabled by default. To enable it, set `debug.enableRawConfig` to true. * `/debug/service_monitors` Returns a JSON list of all dynamic service monitor configurations the Collector tracks. * `/debug/set_log_level` Accepts a `PUT` request to temporarily change the logging level of the Collector for a specified duration. Refer to [temporarily change the Collector log level](/ingest/metrics-traces/collector/configure/optimizations#temporarily-change-the-collector-logging-level) for more information. * `/debug/services` Returns a JSON list of all Kubernetes Services the Collector detects. * `/debug/targets` Returns a JSON list of key-value pairs that represent all Kubernetes targets the Collector monitors. The first element is the Collector configuration, followed by a list of all scrape configurations created from the configuration. The endpoint doesn't display Graphite targets. * `/debug/version` Returns a JSON representation of the Collector version information. This is the same information the Collector displays at startup. ## Debug configuration The following Collector configuration settings control the behavior of debug endpoints. By default, debug endpoints are served only through loopback or localhost. If `debug.exposeDebugPagesOnAllInterfaces` is set to `true`, or if a value is set for `debug.listenAddress`, this restriction is no longer enforced. When this restriction isn't enforced, remote hosts are able to access debug endpoints, which can expose sensitive internal data. ```yaml theme={null} debug: disabled: false exposeDebugPagesOnAllInterfaces: false listenAddress: "" enableRawConfig: false ``` | Setting | Description | Default | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | `debug.disabled` | If `true`, disables all debug endpoints. Requires Collector version 0.117.0 or later. | `false` | | `debug.exposeDebugPagesOnAllInterfaces` | If `true`, debug endpoints are served on all interfaces that the Collector listens to. Chronosphere recommends against enabling this setting in production environments. | `false` | | `debug.listenAddress` | Overrides the default address and port for debug endpoints. This value is distinct from the [global `listenAddress`](/ingest/metrics-traces/collector/configure/optimizations#define-the-listenaddress). | *none* | | `debug.enableRawConfig` | If `true`, enables the `/debug/config_dump` and `/debug/raw_config` endpoints. | `false` | # Configure Chronosphere Collector service discovery for Prometheus metrics Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/discover Compare Chronosphere Collector service discovery methods, including ServiceMonitors, Kubernetes annotations, and Prometheus discovery. Chronosphere Collector supports several mechanisms to discover which metrics applications to scrape Prometheus metrics from: * [ServiceMonitors](#servicemonitors) (each node) * [Kubernetes annotations](#kubernetes-annotations) (each node) * [Prometheus service discovery](#prometheus-service-discovery) (per cluster) Using `ServiceMonitors` or Kubernetes annotations (or a combination of both) are recommended for most deployments. Use push-based collection mechanisms for use cases where jobs can't be scraped automatically, such as AWS Lambda, Google Cloud Functions, or ephemeral batch jobs. ## ServiceMonitors `ServiceMonitors` are a custom resource definition (CRD) you can use to define scrape configurations and options in a separate Kubernetes resource. Discovery is scoped to the targets on the local node by default, which requires you to deploy the Collector as a [DaemonSet](/ingest/metrics-traces/collector/install/kubernetes#install-the-collector) for this method of service discovery. ### Prerequisites Run the following command to install the [`ServiceMonitor` CRD](https://prometheus-operator.dev/docs/getting-started/design/#servicemonitor) from the full Prometheus Operator, using the file in the `kube-prometheus-stack` Helm chart: ```shell theme={null} kubectl apply -f https://raw.githubusercontent.com/prometheus-community/helm-charts/e46dc6360b6733299452c8fd65d304004484de79/charts/kube-prometheus-stack/crds/crd-servicemonitors.yaml ``` Chronosphere supports only fields in version 0.44.1 of the Prometheus Operator. ### Enable ServiceMonitor discovery To enable ServiceMonitor discovery in the Collector, make the following configuration changes: 1. Add the following options after the `ClusterRole` resource in the manifest under `rules`: ```yaml theme={null} kind: ClusterRole rules: - apiGroups: - monitoring.coreos.com resources: - servicemonitors verbs: - get - list - watch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - get - list - watch ``` 2. Enable the `ServiceMonitors` feature of the Collector by setting the following keys to `true` in the manifest `ConfigMap` under the `discovery` > `kubernetes` key: ```yaml theme={null} discovery: kubernetes: enabled: true serviceMonitorsEnabled: true endpointsDiscoveryEnabled: true podMatchingStrategy: VALUE # Use additionalPodMatching only if necessary, such as when using a Virtual Kubelet. # Requires Collector v0.113.0. # additionalPodMatching: # - labelSelector: your-label-with-node-name=${KUBERNETES_NODE_NAME:""} # Dynamically start and stop this additional Pod matcher based # on whether the default Pod matcher identifies any Pods that also # match the given label selector. # activationRule: # podLabelSelector: service=xyz,app=abc ``` * `serviceMonitorsEnabled`: Indicates whether to use ServiceMonitors to generate job configurations. * `endpointsDiscoveryEnabled`: Determines whether to discover Endpoints. Requires `serviceMonitorsEnabled` to be set to `true`. * `podMatchingStrategy`: Determines how to use ServiceMonitors and annotations when discovering targets. Accepts the following settings for *`VALUE`*: * `all`: Allows any and all scrape jobs to be registered for a single pod. * `annotations_first`: Matches annotations first. If no matches return, then other matching can occur. * `service_monitors_first`: Matches ServiceMonitors first. If no matches return, then other matching can occur. * `service_monitors_only`: Matches ServiceMonitors only. * `additionalPodMatching`: An array that configures the discovery of additional Pods in addition to those on the same Node as the Chronosphere Collector Pod, such as when a set of other Pods is divided among the set of Chronosphere Collector DaemonSet Pods. For example, you can configure this to discover and scrape Pods that represent workloads within a [Virtual Kubelet](https://virtual-kubelet.io/) whose host Pod is on the same Node as the Collector Pod. Each item in the `additionalPodMatching` array must specify a `labelSelector` or `fieldSelector` to filter which Pods to watch. Additional Pod matching requires Chronosphere Collector v0.113.0 or later. * `labelSelector`: Matches Pods where a label value matches the name of the Node on which the Collector Pod DaemonSet instance is running. Define the selector value to match additional Pods to be selected. * `activationRule`: Adds an activation rule to `additionalPodMatching` to dynamically start and stop Kubernetes API watches based on the presence of Pods running on the same Node that match the `podLabelSelector`. If a given Collector instance identifies a Pod on its respective Node that matches the activation rule's label selector, that Collector will enable the additional watch stream. If that Pod ceases to exist, the Collector will also stop the additional watch stream. ### Pod-based ServiceMonitor discovery If you use a version of Kubernetes that doesn't support endpoint slices, you can set `endpointsDiscoveryEnabled` to `false` to run the Collector in a mode that doesn't discover Kubernetes endpoint slices or service resources. In this mode, the Collector can still discover scrape targets using `ServiceMonitors` under specific circumstances depending on the Kubernetes resource configuration. The Collector uses the Pod's labels as the Service's labels. If the Pod's labels match the Service's labels a `ServiceMonitor` that uses `targetPort` (container port) to indicate the port to scrapes. Because this discovery method can be very resource intensive, don't use this method for most deployments. Instead, contact [Chronosphere Support](/support) for more information about pod-based ServiceMonitor discovery. #### Run as a DaemonSet with ServiceMonitors If you want to run the Collector as a DaemonSet and scrape `kube-state-metrics` through a Collector running as a Deployment, you need to update the manifest for both Collector instances. In your DaemonSet, add the `serviceMonitor` > `serviceMonitorSelector` key to your manifest and define the following `matchExpressions` to ensure that your DaemonSet only matches on `ServiceMonitors` that don't contain `kube-state-metrics`: ```yaml theme={null} serviceMonitor: serviceMonitorSelector: matchAll: false matchExpressions: - label: app.kubernetes.io/name operator: NotIn values: - kube-state-metrics ``` In your Deployment, add the same key and definitions to your manifest, but set the `operator` value of the `matchExpressions` attribute to `In`. This setting ensures that your Deployment only matches on `ServiceMonitors` that contain `kube-state-metrics`: ```yaml theme={null} serviceMonitor: serviceMonitorSelector: matchAll: false matchExpressions: - label: app.kubernetes.io/name operator: In values: - kube-state-metrics ``` ### Match specific ServiceMonitors By default, the Collector ingests metrics from all `ServiceMonitor` sources. To match specific instances, use a series of `AND` match rules under the `serviceMonitor` > `serviceMonitorSelector` key and set the `matchAll` under the `serviceMonitorSelector` key to `false`. ```yaml theme={null} serviceMonitorSelector: matchAll: false ``` The available match rules are: * `matchLabelsRegexp`: Labels and a regular expression to match a value. For example: ```yaml theme={null} matchLabelsRegexp: labelone: '[a-z]+' ``` * `matchLabels`: Labels and a matching value. For example: ```yaml theme={null} matchLabels: labelone: foo ``` * `matchExpressions`: Depending on the operator set, labels that exist or don't exist, or have or don't have specific values. For example: * To match `ServiceMonitors` that have the `examplelabel` with values `a` **or** `b` use the `In` operator: ```yaml theme={null} matchExpressions: - label: examplelabel operator: In values: - a - b ``` * To match `ServiceMonitors` that have the `examplelabel` without values `a` **or** `b`, use the `NotIn` operator. The `NotIn` operator also matches any `ServiceMonitors` without the `examplelabel` present: ```yaml theme={null} matchExpressions: - label: examplelabel operator: NotIn values: - a - b ``` * To match `ServiceMonitors` that have the `examplelabel` with **any** value, use the `Exists` operator: ```yaml theme={null} matchExpressions: - label: examplelabel operator: Exists ``` * To match `ServiceMonitors` that don't have the `examplelabel`, use the `DoesNotExist` operator: ```yaml theme={null} matchExpressions: - label: examplelabel operator: DoesNotExist ``` ### Match endpoints without pods using ServiceMonitors The default Collector configuration isn't suitable if you want to discover endpoints but lack access to Pod information. For example, if you want to: * Monitor the Kubernetes API server, which doesn't run on the same node as Kubernetes workloads. * Monitor endpoints that can be running anywhere in the cluster, but without using a Collector running as a DaemonSet. * Discover and scrape `kube-state-metrics`, which listen to the Kubernetes API server and generate metrics about deployments, nodes, and pods. If you're monitoring endpoints but don't have access to Pod information, the `ServiceMonitor` can't use the `TargetPort` attribute to target the endpoint and must instead use the `Port` attribute. In these cases, run the Collector as a [Kubernetes Deployment](/ingest/metrics-traces/collector/install/kubernetes#install-the-collector-using-a-deployment) with a single instance, and set the `allowSkipPodInfo` attribute to `true`. ```yaml theme={null} serviceMonitor: allowSkipPodInfo: true ``` Use this attribute with caution. Setting `allowSkipPodInfo` to `true` for a DaemonSet can cause every Collector in the DaemonSet to attempt to scrape every endpoint in the cluster, or cause duplicate scrapes. ### Enable restricted mode > Requires [Chronosphere Collector](/ingest/metrics-traces/collector) version 0.116.0 or later. You can enable restricted mode for ServiceMonitors, which instructs the Collector to ignore certain keys within `serviceMonitor.spec.endpoints` that could allow a malicious ServiceMonitor to access files or URLs accessible by the Collector. The default value is `false`. Chronosphere recommends setting this value to `true` if you run the Collector in an environment where ServiceMonitors might be created by actors who you don't fully trust. ```yaml theme={null} serviceMonitor: restrictedMode: true ``` If restricted mode is enabled, the Collector ignores the following keys within `serviceMonitor.spec.endpoints`: * `BasicAuth` * `BearerTokenFile` * `BearerTokenSecret` * `ProxyURL` * `TLSConfig.{CAFile,CertFile,KeyFile}` ## Kubernetes annotations Discovery is scoped to the targets on the local node by default, which requires you to deploy the Collector as a [DaemonSet](/ingest/metrics-traces/collector/install/kubernetes) for this method of service discovery. For the Collector to start scraping the Pods in a Kubernetes cluster, set the following `annotations` on each Pod in the cluster: ```yaml theme={null} spec: template: metadata: annotations: prometheus.io/scrape: 'true' prometheus.io/port: '{port_number}' ``` The following manifest is an example of using these two annotations for a basic [Node Exporter](https://github.com/prometheus/node_exporter) deployment. Based on these annotations, by default, the Collector starts scraping the `/metrics` endpoint on port `9100`. ```yaml theme={null} apiVersion: apps/v1 kind: DaemonSet metadata: labels: app.kubernetes.io/name: node-exporter app.kubernetes.io/version: v1.0.1 name: node-exporter namespace: monitoring spec: selector: matchLabels: app.kubernetes.io/name: node-exporter template: metadata: annotations: prometheus.io/scrape: "true" prometheus.io/port: "9100" labels: app.kubernetes.io/name: node-exporter app.kubernetes.io/version: v1.0.1 spec: containers: - image: quay.io/prometheus/node-exporter:v1.0.1 name: node-exporter # ... ``` You can set additional annotations to control other scrape options. For a complete list of supported annotations, read the [scrape configuration documentation](/ingest/metrics-traces/collector/discover/scrape-configuration). You can change the annotation prefix, which defaults to `prometheus.io/`, from the `kubernetes` > `processor` section of the Collector `ConfigMap`. After any changes, send the updated manifest to the cluster with the following command: ```shell theme={null} kubectl apply -f path/to/manifest.yml ``` If you modify a Collector manifest, you must [update it in the cluster and restart the Collector](/ingest/metrics-traces/collector/configure#modify-the-collector-manifest). ## Prometheus service discovery If using Prometheus service discovery within Kubernetes, [deploy a single Collector as a Kubernetes Deployment](/ingest/metrics-traces/collector/install/kubernetes#install-the-collector-using-a-deployment) per cluster. This is to avoid every Collector instance duplicating scrapes to all endpoints defined in the Prometheus service discovery configuration. To enable Prometheus service discovery, set `discovery.prometheus.enabled` to `true` in the Collector config. Provide the list of scrape configs in the `discovery.prometheus.scrape_configs` section. The following example uses the [kubernetes\_sd\_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config). ```yaml theme={null} discovery: prometheus: enabled: true scrape_configs: - job_name: kubernetes-pods honor_timestamps: true scrape_interval: 30s scrape_timeout: 30s metrics_path: /metrics scheme: http kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] separator: ; regex: 'true' replacement: $1 action: keep - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] separator: ; regex: (.+) target_label: __metrics_path__ replacement: $1 action: replace - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] separator: ; regex: ([^:]+)(?::\d+)?;(\d+) target_label: __address__ replacement: $1:$2 action: replace - separator: ; regex: __meta_kubernetes_pod_label_(.+) replacement: $1 action: labelmap - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: kubernetes_namespace replacement: $1 action: replace - source_labels: [__meta_kubernetes_pod_name] separator: ; regex: (.*) target_label: kubernetes_pod_name replacement: $1 action: replace ``` For details, see the [Prometheus scrape configuration documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config). For a complete list of examples, see the examples section in the [Prometheus GitHub repository](https://github.com/prometheus/prometheus/tree/main/documentation/examples). ### Set the Collector to scrape its own metrics For the Collector to scrape its own metrics, add another job to the `discovery.prometheus.scrape_configs` key: ```yaml theme={null} # ... - job_name: 'collector' scrape_interval: 30s scrape_timeout: 30s static_configs: - targets: ['0.0.0.0:3030'] # ... ``` # Discover and scrape Kubernetes resources with Chronosphere Collector Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/discover/monitor-kubernetes Configure the Chronosphere Collector to discover and scrape Kubernetes resources, including kubelet and cAdvisor metrics. To monitor Kubernetes resources with Chronosphere Collector and ensure that your clusters are healthy, use one of the following configuration methods. ## Monitor kubelet or cAdvisor metrics If running in Kubernetes, you can configure the Collector to scrape kubelet or cAdvisor metrics by setting the `kubeletMetricsEnabled` or `cadvisorMetricsEnabled` flag to `true` under the `kubeletMonitoring` YAML collection. For example: ```yaml theme={null} discovery: kubernetes: # ... kubeletMonitoring: port: 10250 bearerTokenFile: "/var/run/secrets/kubernetes.io/serviceaccount/token" kubeletMetricsEnabled: true cadvisorMetricsEnabled: true probesMetricsEnabled: true labelsToAugment: [] annotationsToAugment: [] ``` * `port`: The port that the `kubelet` is running on. Default: `10250`. * `bearerTokenFile`: The path to the file containing Collector's service account token. Default: `"/var/run/secrets/kubernetes.io/serviceaccount/token"`. * `kubeletMetricsEnabled`: Enables scraping kubelet metrics. Default: `false`. * `cadvisorMetricsEnabled`: Enables scraping cAdvisor metrics. Default: `false`. * `probesMetricsEnabled`: Enables collecting metrics on the status of liveness, readiness, and startup kubelet probes for Kubernetes containers. Default: `false`. * [`labelsToAugment`](#add-metadata-labels-from-pod-labels): Lists the metadata labels from pod labels that the Collector adds to metrics. * [`annotationsToAugment`](#add-metadata-labels-from-pod-annotations): Lists the metadata labels from pod annotations the Collector adds to metrics. When running the Collector in Kubernetes as a DaemonSet, you can configure the Collector to scrape kubelet or cAdvisor metrics from the node the Collector DaemonSet instance is running on. To configure this behavior, under the `kubeletMonitoring` YAML collection, set the `kubeletMetricsEnabled` or `cadvisorMetricsEnabled` flag to `true`The settings under the `kubletMonitoring` collection are applied only when the Collector is deployed as a DaemonSet. Enabling `kubeletMetricsEnabled`, `cadvisorMetricsEnabled`, or `probesMetricsEnabled` doesn't require the creation of additional ServiceMonitor definitions. Use [global rewrite configurations](/ingest/metrics-traces/collector/discover/scrape-configuration) to manipulate labels for metrics from these sources. ### Add metadata labels from pod labels By default, container-level metrics don't include metadata labels like `service` or `app`, which searches can include when querying for these metrics. To automatically add these labels from pod labels, use the `labelsToAugment` flag to list the labels that Collector adds to the metrics. For example, to add the `app` label to the container level metrics for a `node-exporter` DaemonSet deployment, use the following configuration under the `kubeletMonitoring` key: ```yaml theme={null} discovery: kubernetes: # ... kubeletMonitoring: # ... labelsToAugment: ["app", ...] ``` This adds `app="node-exporter"` to these metrics, based on the following example `node-exporter` manifest: ```yaml theme={null} apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: node-exporter name: node-exporter namespace: monitoring # ... ``` ### Add metadata labels from pod annotations By default, container-level metrics don't include metadata labels, which searches can include when querying for these metrics. To automatically add these labels from pod annotations, use the `annotationsToAugment` flag to list the labels the Collector adds to the metrics. For example, to add the `app_kubernetes_io_component` label to the container-level metrics for a `node-exporter` DaemonSet deployment, use the following configuration under the `kubeletMonitoring` key: ```yaml theme={null} discovery: kubernetes: # ... kubeletMonitoring: # ... annotationsToAugment: ["app.kubernetes.io/component", ...] ``` This adds `app_kubernetes_io_component="infrastructure"` to these metrics, assuming the following example `node-exporter` manifest: ```yaml theme={null} apiVersion: apps/v1 kind: DaemonSet metadata: annotations: app.kubernetes.io/component: infrastructure name: node-exporter namespace: monitoring # ... ``` ## Map Kubernetes labels to Prometheus labels The Collector lets you specify pod labels and annotations you want to keep as a Prometheus label. This feature applies to pods only. The following configuration example converts all pod labels called `my_label` and all pod annotations called `my.pod.annotation` into Prometheus labels for the metrics scraped from discovered pods. This is equivalent to a Prometheus `labelmap` rule, but sanitizes the label names and values: ```yaml theme={null} discovery: kubernetes: enabled: true metadataConfig: - resource: "pod" annotationsToKeep: - my.pod.annotation labelsToKeep: - my_label ``` ## Discover `kube-system` endpoints To discover endpoints in the `kube-system` namespace, set the `kubeSystemEndpointsDiscoveryEnabled` flag to `true`. Because `kube-system` has many constantly changing endpoints that might cause unnecessary load on the Collector, the endpoint is disabled by default. Using EndpointSlices significantly reduces the amount of load on the Kubernetes API server. If you modify a Collector manifest, you must [update it in the cluster and restart the Collector](/ingest/metrics-traces/collector/configure#modify-the-collector-manifest). ## Discover and scrape `kube-state-metrics` You can use `ServiceMonitors` to scrape `kube-state-metrics`, which generate metrics that track the health of deployments, nodes, and pods in a Kubernetes cluster. Monitoring these metrics can help to ensure the health of your cluster because the Collector expects to continually receive `kube-state-metrics`. If the Collector can't scrape these metrics, it's likely your Kubernetes cluster is experiencing issues you need to resolve. Monitoring `kube-state-metrics` with a DaemonSet Collector is manageable for smaller clusters, but can lead to out of memory (OOM) errors as the cluster scales. Chronosphere recommends running the Collector as a sidecar to take advantage of staleness markers. The following steps assume that: * You're running a separate Collector as a Deployment to monitor `kube-state-metrics`. * You've already defined a Kubernetes Service and ServiceMonitor for `kube-state-metrics`. If you're already running the Collector as a DaemonSet, you must update the manifest for both Collector instances. After installing the `ServiceMonitors` CRD, complete the following steps to discover `kube-state-metrics`: 1. Download [this manifest](https://storage.googleapis.com/chronosphere-release/latest/kubernetes/ksm-chronocollector.yaml). 2. In the `data` section, replace the values for `address` and `api-token` with your Base64-encoded API token: ```yaml theme={null} --- apiVersion: v1 data: address: api-token: ``` 3. Apply the manifest: ```shell theme={null} kubectl apply -f path/to/ksm-chronocollector.yaml ``` 4. Confirm the `Deployment` is started and running, and view the logs of the pod. 1. Use `kubectl get pods` to list the pods: ```shell theme={null} kubectl get pods ``` 2. In the output, identify the pod to examine by its `NAME` column value. For example, the pod name in the following output is `chronocollector-jtgfw`. ```text theme={null} NAME READY STATUS RESTARTS AGE chronocollector-jtgfw 1/1 Running 0 1m ... ``` 3. Use `kubectl logs` to review the pod's logs: ```shell theme={null} kubectl logs chronocollector-jtgfw ``` ## Ingest Kubernetes API server metrics The Kubernetes API Server provides REST operations and a frontend to a cluster's shared state through which all other components interact. Unlike most other metrics emitted from a cluster, Kubernetes doesn't expose API Server metrics by using a pod, but instead exposes metrics directly from an endpoint in the API Server. To ingest these metrics through traditional service discovery methods, you must discover and scrape the endpoints directly. The Collector supports using `ServiceMonitors` or job service discovery. ### Discover API Server metrics with ServiceMonitors #### Discover metrics for a managed Kubernetes cluster To use [ServiceMonitors](/ingest/metrics-traces/collector/discover#servicemonitors) to discover and scrape API server metrics from a managed Kubernetes cluster, such as Amazon Elastic Kubernetes Service (EKS) or Google Kubernetes Engine (GKE), enable both the `allowSkipPodInfo` flag under the top level `serviceMonitor` key and the `endpointsDiscoveryEnabled` flag under the `discovery.kubernetes` YAML collection in the Collector configuration. ```yaml theme={null} serviceMonitor: allowSkipPodInfo: true # ... discovery: kubernetes: endpointsDiscoveryEnabled: true ``` If using this method, deploy the Collector as an [individual Kubernetes Deployment](/ingest/metrics-traces/collector/install/kubernetes#install-the-collector-using-a-deployment) rather than a DaemonSet. This prevents multiple Collectors from scraping the same API Server metrics. Additionally, set the appropriate [ServiceMonitor matching](/ingest/metrics-traces/collector/discover#servicemonitors) to prevent other Collectors from discovering the API Server ServiceMonitor. #### Discover metrics for a self-managed Kubernetes cluster To use [ServiceMonitors](/ingest/metrics-traces/collector/discover#servicemonitors) to discover and scrape API server metrics from a self-managed Kubernetes cluster, such as k0ps, enable the `endpointsDiscoveryEnabled` flag under the `discovery.kubernetes` YAML collection in the Collector configuration: ```yaml theme={null} discovery: kubernetes: endpointsDiscoveryEnabled: true ``` In this configuration, you can deploy the Collector as a DaemonSet, which installs the Collector on the master nodes that run the Kubernetes API Server so the Collector can scrape it. ### Discover API Server metrics with the jobs service To discover and scrape API Server metrics without using `ServiceMonitors`, you can use the `jobs` section of the Collector configuration for service discovery. If using this method, deploy the Collector as an [individual Kubernetes Deployment](/ingest/metrics-traces/collector/install/kubernetes#install-the-collector-using-a-deployment) instead of as a DaemonSet. This prevents multiple Collectors from scraping the same API Server metrics. The following is an example that discovers the API Server based on the value of the `__meta_kubernetes_pod_label_k8s_app` label equal to `kube-apiserver` (found in the API Server Service object). ```yaml theme={null} jobs: - name: kube-apiserver selector: kubernetes: matchLabels: __meta_kubernetes_pod_label_k8s_app: kube-apiserver options: scheme: https http: bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token tls: caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecureSkipVerify: true relabels: - sourceLabels: [__name__] targetLabel: service replacement: kube-apiserver - sourceLabels: [__meta_kubernetes_pod_node_name] targetLabel: node ``` # Scrape configuration using Kubernetes Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/discover/scrape-configuration Set global, annotation, and job-level scrape configurations for the Chronosphere Collector. When using Kubernetes discovery, Chronosphere Collector determines the options for scraping endpoints based on this order: 1. Default global scrape configuration 2. [Global scrape configuration](/ingest/metrics-traces/collector/discover/scrape-configuration#global-scrape-configuration) 3. [Kubernetes annotations](/ingest/metrics-traces/collector/discover/scrape-configuration#kubernetes-annotations) 4. [Job configuration](/ingest/metrics-traces/collector/discover/scrape-configuration#define-jobs-configuration) Any settings made by an earlier configuration are overwritten by a later configuration. For example, the default scrape interval as set in the default global scrape configuration is 10 seconds (`10s`). You can override this interval by setting a scrape configuration, adding an annotation to a Kubernetes object, or adding a job configuration. ## Global scrape configuration To apply those settings globally to all jobs, define the following default scrape configuration: ```yaml theme={null} scrape: defaults: metricsPath: "/metrics" scheme: "http" scrapeInterval: 10s scrapeTimeout: 10s honorLabels: false honorTimestamps: true followRedirects: true relabels: metricRelabels: enableCompression: true ``` > The `followRedirects` option requires Chronosphere Collector version 0.117.0 or later. You can override the global `scrape` configuration to set options for your instance. If you don't specify a field, it uses the default values. The [Prometheus `relabel_config`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) configuration applies to `relabels` and `metricRelabels` parameters. This configuration allows for advanced modifications to any target and its labels before Collector ingests metrics. By default, scrapes request gzip compression by using the `Accept-Encoding` header. This is configured by the `enableCompression` option. ## Kubernetes annotations You can use Kubernetes annotations to override any global defaults. The Collector supports the following Prometheus annotations to enable scraping for a pod. By default, `prometheus.io` prefixes each of these annotations, such as `prometheus.io/scrape`. You can [change the annotation prefix](#change-annotation-prefix) to a different value. * `/scrape`: Determines whether to enable scraping. Excludes scraping the pod if set to `false`. * `/port`: The port to scrape on the pod. * `/path`: The pod's metrics path. * `/params`: HTTP URL parameters. * `/scheme`: The connection protocol, either `http` or `https`. * `/job`: Overrides the value of the `job` label. * `/collectionInterval`: Determines how often to scrape endpoints. Default: `10s`. * `/collectionTimeout`: The timeout for a scrape operation. * `/serviceAccountBearerToken`: The token value, if using service account authentication. * `/httpProxyURL`: The Proxy URL of connections to this pod. * `/tlsServerName`: The host name when using TLS authentication. * `/tlsInsecureSkipVerify`: Determines whether to verify a server's certificate. Set to `true` to avoid verifying the server's certificate for validity. The pod being scraped might use some of the following annotations. If so, the Collector pod must also have either an environment variable or a file pointing to `files/env vars`. * `/httpBasicAuthUsernameEnvVar`: The `Username` environment variable when using basic authentication. * `/httpBasicAuthPasswordEnvVar`: The `Password` environment variable when using basic authentication. * `/httpBasicAuthPasswordFile`: The path to a file containing a password when using basic authentication. * `/httpBearerTokenEnvVar`: The `Token` environment variable when using HTTP bearer token authentication. * `/httpBearerTokenFile`: The path to a file containing a bearer token when using HTTP bearer token authentication. * `/tlsCAFile`: The path to a CA certificate when using TLS authentication. * `/tlsCertFile`: The path to a personal certificate when using TLS authentication. * `/tlsKeyFile`: The path to a private key when using TLS authentication. For example, if the Kubernetes pod to be scraped includes the following annotations: ```text theme={null} prometheus.io/scrape=true prometheus.io/port=8090 prometheus.io/httpBasicAuthUsernameEnvVar=MY_USERNAME httpBasicAuthPasswordEnvVar=MY_PASSWORD ``` The Collector pod must define these environment variables to populate the `httpBasicAuthUsernameEnvVar` and `httpBasicAuthPasswordEnvVar` annotations' values: ```text theme={null} MY_USERNAME=admin MY_PASSWORD=mypassword ``` ### Change annotation prefix You can change the annotation prefix by setting the environment variable *`KUBERNETES_PROCESSOR_ANNOTATIONS_PREFIX`* as the value of the `annotationsPrefix` key in the `processor` section of the `kubernetes` collection. The default value is `prometheus.io/`. ```yaml theme={null} kubernetes: # ... # processor defines configuration for processing pods discovered on Kubernetes. processor: # annotationsPrefix is the prefix for annotations that the Collector uses to scrape discovered pods. annotationsPrefix: ${KUBERNETES_PROCESSOR_ANNOTATIONS_PREFIX:"prometheus.io/"} ``` If you modify a Collector manifest, you must [update it in the cluster and restart the Collector](/ingest/metrics-traces/collector/configure#modify-the-collector-manifest). ### Enable restricted mode > Requires [Chronosphere Collector](/ingest/metrics-traces/collector) version 0.116.0 or later. You can enable restricted mode for annotations, which instructs the Collector to ignore certain annotations that could allow a malicious Pod to access files or URLs accessible by the Collector. The default value is `false`. Chronosphere recommends setting this value to `true` if you run the Collector in an environment where Pods might be created by actors who you don't fully trust. ```yaml theme={null} kubernetes: # ... processor: annotationsRestrictedMode: true ``` If restricted mode is enabled, the Collector ignores the following annotations: * `[annotationsPrefix]/httpBasicAuth{Username,Password}EnvVar` * `[annotationsPrefix]/httpBasicAuthPasswordFile` * `[annotationsPrefix]/httpBearerTokenEnvVar` * `[annotationsPrefix]/httpBearerTokenFile` * `[annotationsPrefix]/httpProxyURL` * `[annotationsPrefix]/serviceAccountBearerToken` * `[annotationsPrefix]/tls{CA,Cert,Key}File` If you modify a Collector manifest, you must [update it in the cluster and restart the Collector](/ingest/metrics-traces/collector/configure#modify-the-collector-manifest). ### Scrape multiple ports The `prometheus.io/port` annotation can accept a comma-separated list of the ports to scrape, such as `prometheus.io/port: 1234,5678`. The endpoints must both use the same path. For example both ports `1234` and `5678` must expose Prometheus metrics on the `/metrics` path. If you specify multiple ports, the pods must expose all ports in the list. The following is an example of a container ports configuration in a pod manifest that exposes both `1234` and `5678`: ```yaml theme={null} ports: - name: metrics containerPort: 1234 - name: other_metrics containerPort: 5678 ``` ### Define `jobs` configuration When using annotations for scraping, you can configure the `jobs` collection to override any of the defaults set in the global scrape configuration. The following example contains the value types for the `jobs` collection: ```yaml theme={null} jobs: - name: "example_job" options: metricsPath: PATH params: STRING scheme: SCHEME scrapeInterval: SCRAPE_INTERVAL scrapeTimeout: SCRAPE_TIMEOUT honorLabels: BOOLEAN honorTimestamps: BOOLEAN followRedirects: BOOLEAN relabels: RELABEL_CONFIG metricRelabels: RELABEL_CONFIG enableCompression: BOOLEAN ``` The `name` value must be unique across all scrape configurations, and must match the job name set in the annotations. For example, `prometheus.io/job`. For example, to change the `scrapeInterval` to `1m` and create a relabeling rule that changes the name of the label `code` to `status_code` for the `api` job, add the following configuration to the `config.yml` field of the `chronocollector-config` ConfigMap: ```yaml theme={null} jobs: - name: api options: scrapeInterval: 1m relabels: - action: replace regex: (.*) replacement: $1 sourceLabels: - code targetLabel: status_code ``` If you modify a Collector manifest, you must [update it in the cluster and restart the Collector](/ingest/metrics-traces/collector/configure#modify-the-collector-manifest). ### Differences between `relabel` and `metricRelabels` A Prometheus `relabel` configuration defines the targets you want to scrape and the target labels. Using `relabel` rewrites the label set of a target before it's scraped. Collector applies `metricRelabels` after the scrape but before ingesting data. Use `metricRelabels` if there are expensive metrics that you want to drop, or labels coming from the scrape itself that you want to manipulate. In this example, the `relabel` rule replaces the name of the label `code` to `status_code` for the `api` job before scraping the metrics: ```yaml theme={null} jobs: - name: api options: scrapeInterval: 1m relabels: - action: replace regex: (.*) replacement: $1 sourceLabels: - code targetLabel: status_code ``` The following `metricsRelabel` rule replaces the `pod` and `job` labels for the `cadvisor` job on `promremotebench-0` to `database`: ```yaml theme={null} jobs: - name: cadvisor options: metricRelabels: - action: replace regex: promremotebench-0;(.*) replacement: $1 sourceLabels: - pod - job targetLabel: database ``` [Environment variable expansions](/ingest/metrics-traces/collector/configure/optimizations#implement-environment-variables) and [Prometheus relabel rule](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) regular expression capture group references can use the same syntax. For example, `${1}` is valid in both contexts. If your relabel configuration uses Prometheus relabel rule regular expression capture group references, and they are in the `${1}` format, escape the syntax by adding an extra `$` character to the expression such as `$${1}`. #### `metricRelabels` actions The following actions are available for `metricRelabels`: * `replace`: Match a regular expression against the concatenated `sourceLabels`. Then, set `targetLabel` to `replacement`, with match group references (such as `${1}` or `${2}`) in the replacement substituted by their value. If the regular expression doesn't match, no replacement occurs. * `keep`: Drop targets for which a regular expression doesn't match the concatenated `sourceLabels`. ```yaml theme={null} jobs: - name: kubelet options: metricRelabels: - action: keep regex: (?i)(kubelet_volume_stats_available_bytes|kubelet_volume_stats_capacity_bytes) sourceLabels: [__name__] ``` * `drop`: Drop targets for which the regular expression matches the concatenated `sourceLabels`. The following example demonstrates using `relabels` to drop the `__name__` on `cadvisor` metrics: ```yaml theme={null} jobs: - name: cadvisor options: relabels: - action: drop regex: (.*) sourceLabels: [__name__] ``` * `labelmap`: Match a regular expression against all label names. Then, copy the values of the matching labels to the label names given by the replacement with match group references (such as `${1}` or `${2}`) in the replacement substituted by their value. The following example uses `labelmap` to copy all labels containing `__meta_kubernetes_service_label_` and keep only a portion: ```yaml theme={null} - action: labelmap regex: __meta_kubernetes_service_label_(.+) ``` In this example, the label `__meta_kubernetes_service_label_app='api'` is changed to `app='api'`. * `labeldrop`: Match a regular expression against all label names and remove any label that from the set of labels. The following example matches all label names that contain `container_label_com_amazonaws_ecs_task_arn`: ```yaml theme={null} metricRelabels: - regex: 'container_label_com_amazonaws_ecs_task_arn' action: labeldrop ``` * `labelkeep`: Match a regular expression against all label names and remove any label that doesn't match from the set of labels. The following example drops all label names that don't match `job`: ```yaml theme={null} metricRelabels: - regex: 'job' action: labelkeep ``` # Install the Chronosphere Collector Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/install Compare Chronosphere Collector installation methods for collecting metrics with Kubernetes or a standalone binary and for receiving traces. The Chronosphere Collector can either collect metrics data or receive trace data. When collecting metrics data, the Collector *pulls* data it scrapes from Prometheus endpoints. When receiving trace data, an app *pushes* data to a service, with the Collector behind that service receiving the data. Regardless of the installation method you use, Chronosphere recommends deploying separate, independent Collectors for metrics and for traces. This distinction means that a Collector either collects metrics data or receives trace data. ## Collect metrics data You can install the Collector to retrieve metrics data, either with Kubernetes or as a standalone binary. When deployed within Kubernetes, the Collector scrapes Prometheus endpoints exposed on pods, based on pod-level annotations. ### With Kubernetes You can install the Collector with Kubernetes to collect metrics as: * [A Kubernetes DaemonSet](/ingest/metrics-traces/collector/install/kubernetes) (recommended) * [A Deployment](/ingest/metrics-traces/collector/install/kubernetes#install-the-collector-using-a-deployment) * [A sidecar](/ingest/metrics-traces/collector/install/kubernetes#install-the-collector-as-a-sidecar) Chronosphere recommends installing the Collector as a Kubernetes DaemonSet on each node in a cluster to: * Minimize cross-node network traffic within a cluster. * Eliminate a single point of failure in case one node is offline. * Provide the ability to switch to a standalone deployment if the Collector is using too many resources. ### Without Kubernetes If you don't use Kubernetes, or want to gather metrics from services not managed by Kubernetes, you can install the Collector [as a standalone binary](/ingest/metrics-traces/collector/install/standalone). Docker images are `multiplatform` builds for both `linux/amd64` and `linux/arm64`. Chronosphere also offers [Linux binaries](/ingest/metrics-traces/collector/install/standalone#download-the-collector-binary-and-make-it-executable) for both of these platforms. ## Receive trace data To receive trace data, Chronosphere recommends [installing the Collector](/ingest/metrics-traces/collector/install/tracing) as a Kubernetes Deployment, typically with a service in front to receive trace data. Your tracing app pushes trace data to the service, which the Collector receives the data from. # Kubernetes Collector installation to retrieve metrics Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/install/kubernetes Install the Chronosphere Collector as a Kubernetes DaemonSet, Deployment, or sidecar to scrape Prometheus metrics from your cluster. The recommended way to install the Chronosphere Collector to retrieve metrics data is with a Kubernetes DaemonSet scraping endpoints, using either annotations or ServiceMonitors. This configuration ensures that if a Collector pod restarts or terminates, only the metrics in that node aren't scraped, and all other nodes remain unaffected. The following steps align with this method of installing the Collector. You can also install the Collector with Kubernetes [as a Deployment](/ingest/metrics-traces/collector/install/kubernetes#install-the-collector-using-a-deployment) or [as a sidecar](/ingest/metrics-traces/collector/install/kubernetes#install-the-collector-as-a-sidecar). The following steps assume that the applications running in the Kubernetes cluster expose Prometheus metrics. To ingest other metric types such as StatsD, refer to [Ingest additional metrics](/ingest/metrics-traces/collector/addl-metrics). To install the Collector: 1. [Create an API token](#create-an-api-token). 2. [Define the Collector manifest](#define-the-collector-manifest). 3. [Create a Kubernetes secret](#create-a-kubernetes-secret). 4. [Install the Collector](#install-the-collector). 5. [Verify the Collector](#verify-the-collector). ## Create an API token To interact with the Collector, you must [create a service account](/administer/accounts-teams/service-accounts#create-a-restricted-service-account). Chronosphere recommends creating a restricted service account with a write-only scope. Use the generated API token in your Kubernetes `Secret` to authenticate with the Collector. Store your API token in a secure location. If you lose your token, you must create a new service account. ## Define the Collector manifest 1. Download the [example manifest](https://storage.googleapis.com/chronosphere-release/latest/kubernetes/chronocollector.yaml). 2. Locate the `spec.template.spec.containers.image` YAML collection and change `vVERSION_NUMBER` to the version of the Collector you want to install. For example, `v0.116.0`: ```yaml theme={null} spec: template: spec: containers: - image: gcr.io/chronosphereio/chronocollector:vVERSION_NUMBER ``` Chronosphere supports Chronosphere Collector versions for a year from release. You can find a full list of versions and release dates in the Collector release notes located in the Chronosphere Observability Platform. To view the release notes, in the navigation menu select ** More information > Release notes**, and then click the **Collector** tab. 3. Modify any configuration values in the `ConfigMap` based on your needs. Refer to the [Collector configuration](/ingest/metrics-traces/collector/configure) for details about configuration options. ## Create a Kubernetes secret Create a Kubernetes `Secret` that includes the API token from your service account, plus a custom domain address. 1. Add `base64` encoding to your domain name and API token: ```shell theme={null} echo -n "TENANT:443" | base64 echo -n "API_TOKEN" | base64 ``` Replace the following: * * 2. Copy the values that output to your terminal, which you add as part of your Kubernetes `Secret` in the next step. 3. In your `chronocollector.yaml` file, locate the following YAML collection and replace values for `address` and `api-token` with the encoded values that you copied from your terminal. ```yaml theme={null} apiVersion: v1 data: address: ENCODED_ADDRESS api-token: ENCODED_API_TOKEN kind: Secret metadata: labels: app: chronocollector name: chronosphere-secret namespace: default type: Opaque ``` Another option is to add the *`ADDRESS`* and *`API_TOKEN`* directly to the Kubernetes cluster using `kubectl`: ```shell theme={null} kubectl create secret generic chronosphere-secret \ --from-literal=api-token=API_TOKEN \ --from-literal=address=ADDRESS ``` 4. Save and close your `chronocollector.yaml` file. Now you're ready to [install the Collector](#install-the-collector). ## Install the Collector To install the Collector, apply the `chronocollector.yaml` manifest that you configured. When using a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/), you must install the Collector on each node in a cluster. Installing the Collector grants permissions to read metadata about pods scheduled on the local node. Review your `chronocollector.yaml` manifest carefully before applying it. 1. Apply the manifest. ```shell theme={null} kubectl apply -f path/to/chronocollector.yaml ``` 2. Confirm that the `DaemonSet` is up and running, and view the logs of the pod: ```shell theme={null} kubectl get pods NAME READY STATUS RESTARTS AGE chronocollector-POD_NAME 1/1 Running 0 1m ``` 3. View the pod logs to ensure there are no errors: ```shell theme={null} kubectl logs chronocollector-POD_NAME # ... ``` Replace *`POD_NAME`* with the name of the Kubernetes pod where your Collector instance is running. ## Verify the Collector After installing the Collector, verify it's sending metrics to your Observability Platform tenant: 1. Open Observability Platform and in the navigation menu select ** Explorers > Metrics Explorer**. 2. Enter the following query in the query field. ```text theme={null} count(chronocollector_jobs) by (instance) ``` 3. Click **Run**. The name of your Collector instance returned from the `kubectl logs` command displays in the table of metrics: | Time | Instance | | ------------------- | --------------------------------- | | 2023-02-03 10:56:04 | default/chronocollector-POD\_NAME | For other methods of verifying that the Collector is running, refer to [verify and monitor the Collector](/ingest/metrics-traces/collector/monitor). ### Install the Collector using a Deployment If you use Prometheus service discovery, deploy the Collector as a standalone Deployment. This implementation avoids every Collector duplicating scrapes to all endpoints defined in the Prometheus service discovery configuration. Only run the Collector as a [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) if you're using configuration-based Prometheus service discovery, or where specified in the [service discovery documentation](/ingest/metrics-traces/collector/discover). Otherwise, deploy the Collector as a DaemonSet. The instructions for deploying the Collector as a Deployment are the same as for a DaemonSet, except you use [this manifest](https://storage.googleapis.com/chronosphere-release/latest/kubernetes/deployment-chronocollector.yaml). ### Install the Collector as a sidecar Some environments (like AWS Fargate) restrict users from deploying an app as a `DaemonSet` and require deploying the Collector as a sidecar alongside other containers. You can also use a sidecar configuration for more control over resource utilization for particularly high cardinality scrape targets. If your app is already running as a container in a Pod, you can add the Collector as another container in the same Pod. The instructions for deploying the Collector as a sidecar are the same as for a `DaemonSet`, except you use [this manifest](https://storage.googleapis.com/chronosphere-release/latest/kubernetes/sidecar-chronocollector.yaml) and add the Collector as another container to the `spec.template.spec.containers` block of the manifest, along with any other Kubernetes resources your app requires. ## Configure service discovery After configuring and installing the Collector, [configure Kubernetes annotations](/ingest/metrics-traces/collector/discover#kubernetes-annotations) so that the Collector can start scraping the Pods in your Kubernetes cluster. If you installed the Collector as a Deployment or as a sidecar service, [configure service discovery](/ingest/metrics-traces/collector/discover) using `ServiceMonitors` or Prometheus service discovery. ## Modify Collector resource usage After deploying the Collector, you can modify resource usage depending on various inputs, including, but not limited to: * Amount of scraped metrics * Scrape interval of each target * Number of metric labels * Total length of the label strings Because the work of the Collector is I/O bound, Chronosphere recommends running the Collector with one [Kubernetes CPU unit](https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#cpu-units) or its equivalent. To simplify the process of predicting memory usage, make an initial estimation based on the number of metrics scraped every second by your Collector instance. The following table lists the recommended memory limits for a Collector instance, with its default settings based on observed usage: This information is both a starting point and an upper limit. Tune your Collectors after their deployments, based on practical usage. | Metrics/Second | Recommended memory | | ---------------- | --------------------------------------------- | | 0k to 10k | 500 MB | | 10k to 20k | 800 MB | | 20k to 30k | 1 GB | | Greater than 30k | 1 GB and 200 MB per additional 5k metrics/sec | ### Ingestion buffering Enabling [ingestion buffering](/ingest/metrics-traces/collector/configure/optimizations#enable-ingestion-buffering) doesn't have any substantial impact on memory usage. The following graph displays the memory usage of a Collector instance handling different input data volume (4k and 3k metrics per second, respectively). The memory usage doesn't change significantly after enabling the buffering feature or increasing the failure rate from 0% to 50%. Graph showing effect of buffering on memory usage # Standalone Collector installation to retrieve metrics Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/install/standalone Run the Chronosphere Collector as a standalone binary to gather metrics from services that Kubernetes doesn't manage. If you don't use Kubernetes, or want to gather metrics from services not managed by Kubernetes, you can run the Collector as a standalone binary. ## Create an API token To interact with the Collector, you must [create a service account](/administer/accounts-teams/service-accounts#create-a-restricted-service-account). You must be a member of a team with the SysAdmin role to create a new service account. Chronosphere recommends creating a restricted service account with a write-only scope. Use the generated API token in your Kubernetes `Secret` to authenticate with the Collector. Store your API token in a secure location. If you lose your token, you must create a new service account. ## Download the Collector binary and make it executable Chronosphere supports Chronosphere Collector versions for a year from release. You can find a full list of versions and release dates in the Collector release notes located in the Chronosphere Observability Platform. To view the release notes, in the navigation menu select ** More information > Release notes**, and then click the **Collector** tab. 1. Download one of the platform-specific binaries: * [Linux amd64](https://storage.googleapis.com/chronosphere-release/latest/chronocollector-linux-amd64) * [Linux arm64](https://storage.googleapis.com/chronosphere-release/latest/chronocollector-linux-arm64) * [Windows x86-64](https://storage.googleapis.com/chronosphere-release/latest/chronocollector-windows-amd64.exe) * [macOS](https://storage.googleapis.com/chronosphere-release/latest/chronocollector-darwin-amd64) 2. Modify the permissions on the binary to make it executable. For example, to change permissions on the `linux-amd64` version of the binary: ```shell theme={null} chmod +x chronocollector-linux-amd64 ``` ## Define the Collector configuration file Download the [example configuration file](https://storage.googleapis.com/chronosphere-release/latest/standalone/config.yml) and modify any configuration values based on your needs. For applicable fields, you can set environment variables instead of specifying them in the configuration file. For example, to specify the `listenAddress`, set an environment variable named `${LISTEN_ADDRESS}` with the value you want to use, such as: ```yaml theme={null} listenAddress: "${LISTEN_ADDRESS:0.0.0.0:3029}" ``` The following list includes key fields for which you can specify values: * **Backend**: You must add a `gateway` backend to specify your Chronosphere instance. ```yaml theme={null} backend: type: gateway gateway: address: TENANT:443 insecure: false cert: "" certSkipVerify: false ``` * **Global Labels**: If you don't need to apply global labels to all metrics, remove the `labels` key. If you need global labels and are scraping metrics from a Prometheus endpoint, you can apply labels to each scrape target. For example, to add `host` and `rack` to each metric: ```yaml theme={null} static_configs: - targets: ['0.0.0.0:9100'] labels: host: 'prod-server' rack: '14' ``` Each Prometheus scrape configuration type has a different way of specifying global labels. Refer to the Prometheus [scrape configuration documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) for more details. * **Discovery**: To scrape a static Prometheus endpoint, configure a scrape setting. Refer to the Prometheus [service discovery documentation](/ingest/metrics-traces/collector/discover) for more details. ## Run the Collector You can run the Collector standalone with a binary, with Docker, or as a Systemd service. To run the Collector, pass the configuration file and generated API token. Replace the following values before running the startup commands: * * * *`VERSION`*: Version of the Collector that you want to run, prefixed with `v`. ### Run standalone with a binary ```shell theme={null} GATEWAY_ADDRESS=TENANT:443 API_TOKEN=API_TOKEN PATH_TO_FILE/COLLECTOR_BINARY -f PATH_TO_FILE/config.yml ``` ### Run standalone with Docker ```shell theme={null} docker run -v $(pwd)/PATH_TO_FILE/config.yml:/etc/chronocollector/config.yml -e "GATEWAY_ADDRESS=TENANT:443" -e "API_TOKEN=API_TOKEN" gcr.io/chronosphereio/chronocollector:vVERSION ``` ### Run standalone as a service You can install the Collector as a Systemd service on Linux systems: 1. Create a `chronocollector` user and group: ```shell theme={null} sudo groupadd -f chronocollector sudo useradd -g chronocollector --no-create-home --shell /bin/false chronocollector ``` 2. Change the ownership of the directory containing the Collector configuration file. The following command expects the Collector `config.yml` file in the `/etc/chronocollector` directory: ```shell theme={null} sudo chown -R chronocollector:chronocollector /etc/chronocollector ``` 3. Copy the Collector binary to the `/usr/bin` directory and modify the permissions: ```shell theme={null} sudo cp ./chronocollector /usr/bin/ sudo chown chronocollector:chronocollector /usr/bin/chronocollector ``` 4. In the `/usr/lib/systemd/system` directory, create and define a script such as `chronocollector.service` for the Collector service: ```text theme={null} [Unit] Description=Chronocollector Documentation=https://docs.chronosphere.io/ingest/metrics-traces/collector Wants=network-online.target After=network-online.target [Service] User=chronocollector Group=chronocollector Type=simple Restart=on-failure Environment="GATEWAY_ADDRESS=MY_COMPANY.chronosphere.io:443" Environment="API_TOKEN=API_TOKEN" ExecStart=/usr/bin/chronocollector -f /etc/chronocollector/config.yml [Install] WantedBy=multi-user.target ``` 5. Change the permissions of the `chronocollector.service` script you created: ```shell theme={null} sudo chmod 664 /usr/lib/systemd/system/chronocollector.service ``` 6. Enable and verify the `chronocollector` service: ```shell theme={null} sudo systemctl daemon-reload sudo systemctl start chronocollector sudo systemctl status chronocollector sudo systemctl enable chronocollector.service ``` # Kubernetes Collector installation to receive traces Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/install/tracing Install the Chronosphere Collector as a Kubernetes Deployment to receive OpenTelemetry, Jaeger, or Zipkin trace data from your apps. The recommended way to install the Collector to receive trace data is with a Kubernetes Deployment, with a service in front to receive trace data. Your tracing app pushes trace data to the service, which the Collector receives the data from. Chronosphere recommends instrumenting your app to emit span data in [OpenTelemetry format](https://opentelemetry.io/docs/reference/specification/protocol/otlp/) if you haven't already invested in legacy tracing instrumentation. You can configure the Collector to accept OpenTelemetry traces in both gRPC and JSON over HTTP formats. A single Collector can accept OpenTelemetry traces in both of these formats simultaneously. The Collector manifest linked in this page uses OpenTelemetry by default, but you can modify the manifest to use other supported formats, such as Jaeger or Zipkin. To install the Collector: 1. [Create an API token](#create-an-api-token). 2. [Define the Collector manifest](#define-the-collector-manifest). 3. [Create a Kubernetes secret](#create-a-kubernetes-secret). 4. [Install the Collector](#install-the-collector). 5. [Verify the Collector](#verify-the-collector). ## Create an API token To interact with the Collector, you must [create a service account](/administer/accounts-teams/service-accounts#create-a-restricted-service-account). You must be a member of a team with the SysAdmin role to create a new service account. Chronosphere recommends creating a restricted service account with a write-only scope. Use the generated API token in your Kubernetes `Secret` to authenticate with the Collector. Store your API token in a secure location. If you lose your token, you must create a new service account. ## Define the Collector manifest 1. Download the [example manifest](https://storage.googleapis.com/chronosphere-release/latest/kubernetes/tracing-chronocollector.yaml). 2. Locate the `spec.template.spec.containers.image` YAML collection and change `vVERSION_NUMBER` to the version of the Collector you want to install. For example, `v0.116.0`: ```yaml theme={null} spec: template: spec: containers: - image: gcr.io/chronosphereio/chronocollector:vVERSION_NUMBER ``` Chronosphere supports Chronosphere Collector versions for a year from release. You can find a full list of versions and release dates in the Collector release notes located in the Chronosphere Observability Platform. To view the release notes, in the navigation menu select ** More information > Release notes**, and then click the **Collector** tab. 3. Define the format you want the Collector to receive spans in, such as OpenTelemetry, Jaeger or Zipkin. The following example enables receiving spans in OpenTelemetry format: You can only enable one format of traces to receive. You must remove or comment out any additional trace formats from the `spans` YAML collection. 1. In the `spans` YAML collection, define the format you want to receive spans for. For example, to receive spans in OpenTelemetry format: ```yaml theme={null} spans: enabled: true compression: enabled: true otel: enabled: true listenAddress: 0.0.0.0:4317 ``` If you want to accept OpenTelemetry traces in JSON format over HTTP, add the following `http` definitions to your YAML collection under `spans`. This YAML collection starts an HTTP receiver that accepts traces in both gRPC and JSON over HTTP formats. ```yaml theme={null} spans: # ... otel: enabled: true listenAddress: 0.0.0.0:4317 httpEnabled: true httpListenAddress: 0.0.0.0:4318 ``` 2. For the tracing `Service`, modify the `spec.ports.name` YAML collection to reflect the trace format you want to use. For example, to receive spans in OpenTelemetry format: ```yaml theme={null} kind: Service # ... spec: ports: - name: otel port: 4317 protocol: TCP targetPort: otel ``` 4. Save your Collector manifest, which you apply in a subsequent step after creating a Kubernetes secret. ## Create a Kubernetes secret Create a Kubernetes `Secret` that includes the API token from your service account, plus a custom domain address. 1. Add `base64` encoding to your domain name and API token: ```shell theme={null} echo -n "TENANT:443" | base64 echo -n "API_TOKEN" | base64 ``` Replace the following: * * 2. Copy the values that output to your terminal, which you add as part of your Kubernetes `Secret` in the next step. 3. In your `tracing-chronocollector.yaml` file, locate the following YAML collection and replace values for `address` and `api-token` with the encoded values that you copied from your terminal. ```yaml theme={null} apiVersion: v1 data: address: ENCODED_TENANT api-token: ENCODED_API_TOKEN kind: Secret metadata: labels: app: chronocollector-tracing name: chronosphere-tracing-secret namespace: default type: Opaque ``` Another option is to add the *`TENANT`* and *`API_TOKEN`* directly to the Kubernetes cluster using `kubectl`: ```shell theme={null} kubectl create secret generic chronosphere-secret \ --from-literal=api-token=API_TOKEN \ --from-literal=address=TENANT ``` 4. Save and close your `tracing-chronocollector.yaml` file. Now you're ready to [install the Collector](#install-the-collector). ## Install the Collector To install the Collector, apply the `tracing-chronocollector.yaml` manifest that you configured. Installing the Collector grants permissions to read metadata about pods scheduled on the local node. Review your `tracing-chronocollector.yaml` manifest carefully before applying it. 1. Apply the manifest. ```shell theme={null} kubectl apply -f path/to/tracing-chronocollector.yaml ``` 2. Confirm the deployment is up and running, and view the logs of the pod: ```shell theme={null} kubectl get pods ``` The output is similar to the following: ```text theme={null} NAME READY STATUS RESTARTS AGE chronocollector-tracing-POD_NAME 1/1 Running 0 1m ``` 3. View the pod logs to ensure there are no errors: ```shell theme={null} kubectl logs chronocollector-tracing-POD_NAME ``` Replace *`POD_NAME`* with the name of the Kubernetes pod where your Collector instance is running. ## Verify the Collector After installing the Collector, verify that it's sending traces to your Observability Platform tenant. 1. In the navigation menu, click ** Go to Admin** and then select ** Analyzers > Live Telemetry**. 2. Click the **Traces** tab. 3. Click **Capture live data** to display the stream of incoming spans. The default grouping shows spans by service (`__service__`), but you can include additional tags to group by such as `__trace_id__` and `__span_id__`. # Upgrade the Collector Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/install/upgrade Upgrade the Chronosphere Collector to a new version. When you're ready to upgrade the Chronosphere Collector to a new version, determine your current version before upgrading. You can then upgrade the Collector based on your installation method. Chronosphere supports Collector versions for one year from their release dates. Use only a supported version, unless explicitly instructed otherwise by Chronosphere Support. If you experience issues, upgrade to the latest supported version. ## Determine your current version Use one of the following methods to determine the current version of your Collector: * [Check your manifest](#check-your-manifest) * [Query the `/debug/version` endpoint](#query-the-/debug/version-endpoint) * [Use Metrics Explorer](#use-metrics-explorer) ### Check your manifest Open your Collector manifest and locate the `spec.template.spec.containers.env.image` YAML collection. The string includes `vVERSION`, where `VERSION` is the Collector version number: ```yaml theme={null} spec: # ... template: spec: containers: - env: image: gcr.io/chronosphereio/chronocollector:vVERSION ``` *`VERSION`* is the current version number of the Collector. ### Query the `/debug/version` endpoint The Collector exposes a `/debug/version` endpoint that you can query to return the Collector version. To learn about the supported methods for accessing debug endpoints, see [Access the debug endpoints](/ingest/metrics-traces/collector/debugging#access-the-debug-endpoints). The query response returns a JSON representation of the Collector version information, which is the same information the Collector prints when starting. For example: ```json theme={null} { "go_version": "go1.23.4", "version": "v0.116.0", "branch": "HEAD", "revision": "a4c82d1f7", "build_date": "2026-05-12-18:42:05", "build_time": "1778876525" } ``` ### Use Metrics Explorer 1. In the navigation menu select ** Explorers > Metrics Explorer** to access the Metrics Explorer. 2. Enter the following query in the query field. ```text theme={null} count(chronocollector_build_information) by (build_version, instance) ``` 3. Click **Run**. The version of each Collector instance displays in the `build_version` column of the query results table. For example: | Time | build\_version | instance | | ------------------- | -------------- | ------------------------------------ | | 2025-10-10 11:07:15 | v0\_116\_0 | test-env/chronocollector-test3-2z8dx | | 2025-10-10 11:07:15 | v0\_115\_0 | test-env/chronocollector-test3-4vcd9 | ## Upgrade to a new version After determining your current version, you're ready to upgrade your Collector. Your installation method determines how you upgrade the Collector. ### Upgrade a Kubernetes deployment If you deployed your Collector as a Kubernetes DaemonSet or Deployment, complete the following steps to upgrade your Collector. 1. Open your Collector manifest and locate the `spec.template.spec.containers.env.image` YAML collection. 2. In the `image` string, modify the version number to the version that you want to upgrade to. For example, the following `image` indicates version `0.116.0` of the Collector: ```yaml theme={null} spec: # ... template: spec: containers: - env: image: gcr.io/chronosphereio/chronocollector:v0.116.0 ``` 3. Complete a rolling restart of your pods: ```shell theme={null} kubectl rollout restart daemonset chronocollector -n NAMESPACE ``` Replace *`NAMESPACE`* with your Kubernetes namespace. 4. Re-enable port forwarding for your Collector. The following command enables port forwarding on port `3030`: ```shell theme={null} kubectl port-forward chronocollector-POD_NAME 3030 ``` ### Upgrade a standalone deployment If you deployed your Collector as a standalone binary, download the latest version and redeploy the Collector. Refer to the [Standalone Collector installation](/ingest/metrics-traces/collector/install/standalone) page for steps on how to download the latest binary and redeploy it. # Map data sources to Chronosphere Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/mappings Map metrics data and query functions from Prometheus, Datadog, SignalFx, and Wavefront to Chronosphere Observability Platform. To display metrics and traces to users, Chronosphere ingests data through the use of its own [Collector](/ingest/metrics-traces/collector). The Collector supports several metrics formats. Use this section to learn more about how to map other metrics formats' data and functions to Chronosphere: * [Prometheus](/ingest/metrics-traces/collector/mappings/prometheus/prometheus-recommendations) is the default for Chronosphere. * [Datadog](/ingest/metrics-traces/collector/mappings/datadog) data can be imported into Chronosphere. * [SignalFx](/ingest/metrics-traces/collector/mappings/signalfx) queries work differently than PromQL. * [Wavefront](/ingest/metrics-traces/collector/mappings/wavefront) queries work differently than PromQL. # Datadog migration information Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/mappings/datadog Learn how Chronosphere Observability Platform ingests metrics, and plan a migration from Datadog Agent and DogStatsD. Datadog is a monitoring and analytic tool for measuring system performance metrics and event monitoring. The custom metrics are ingested into Datadog platform using locally installed components, such as: * Datadog Agent, which collects metrics through integration. * DogStatsD is a UDP server listening for applications to send the metrics based on a push model. [Learn more about ingesting DogStatsD](/ingest/metrics-traces/collector/addl-metrics/dogstatsd). * Using the HTTP API. Datadog Agent and DogStatsD are lightweight software designed to get metrics and forwarding them to the Datadog back end for persistence. As metrics data increases in scale, the cost of ingress traffic and data persistence can increase exponentially and become difficult to control. Chronosphere Observability Platform approaches monitoring and analysis differently. The Collector ingests metrics and traces using a pull model, sending data to the Observability Platform back end. The Collector is compliant with the major open source standards, including Prometheus, OpenTelemetry, and older formats (like Graphite/StatsD). The Observability Platform control layer determines how metrics and traces should be aggregated, dropped, or rolled-up. Businesses can take advantage of these features to save costs and improve monitoring performance. Many of the Datadog constructs translate to Observability Platform. Although the Customer Success team handles the initial migration from Datadog to Observability Platform, use the following pages to understand how each product approaches data: * [Migration considerations](/ingest/metrics-traces/collector/mappings/datadog/migration-issues): What you need to know before you migrate. * [Querying metrics](/ingest/metrics-traces/collector/mappings/datadog/dogstatsd): Understanding how Observability Platform differs from the Datadog query syntax. * [Monitor migration](/ingest/metrics-traces/collector/mappings/datadog/convert-monitors): Understanding how Observability Platform monitors differ from Datadog monitors. * [Mapping](/ingest/metrics-traces/collector/mappings/datadog/mapping): A breakdown of how various functions and operations work in each product, and how they correspond. * [Widgets versus dashboards](/ingest/metrics-traces/collector/mappings/datadog/dashboards): Learn how some Datadog widgets relate to Observability Platform dashboards. Migrating from Datadog to Observability Platform follows these general steps: 1. Review the ingestion model. 2. Migrate metric collection. 3. Migrate dashboards. 4. Migrate monitors and alerts. 5. Optimize data ingestion and persisted data. # Convert Datadog monitors Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/mappings/datadog/convert-monitors Convert a Datadog monitor into the equivalent monitor, notification policy, and notifier resources in Chronosphere Observability Platform. [Datadog monitors](https://docs.datadoghq.com/api/latest/monitors/) actively check metrics of the infrastructure and manage alerts on alert platforms. Chronosphere metrics of the infrastructure and manage alerts on alert platforms. Chronosphere Observability Platform uses [monitors and alerts](/investigate/alerts) for the same purposes. Datadog creates monitoring and notification in one longer file, while Chronosphere separates monitors and notifications into smaller logical configuration files. These smaller files enable users to target and update specific changes without risking the entire configuration. ## Before you begin A single Datadog monitor becomes several Observability Platform resources: a `Monitor`, a `NotificationPolicy`, and one or more `Notifier` definitions. Before you convert a monitor, gather the following details from the Datadog definition so you can map them to the correct resource: * **Query**: The Datadog `query` becomes the `Monitor` PromQL query. Review [querying metrics](/ingest/metrics-traces/collector/mappings/datadog/dogstatsd) for how the query syntax differs. * **Thresholds and severities**: The Datadog `options.thresholds` (critical, warning) and any no-data behavior become `series_conditions` on the `Monitor`. Observability Platform doesn't support separate recovery thresholds. * **Notification routes**: Each `@`-mention in the Datadog `message` (such as a Slack channel or PagerDuty service) becomes a `Notifier`, and the routing logic becomes a `NotificationPolicy`. * **Tags**: Datadog `tags` become `Monitor` labels, which you can then use to route alerts with notification policy overrides. Review [migration considerations](/ingest/metrics-traces/collector/mappings/datadog/migration-issues) before you start, then use the [field mapping](#field-mapping) tables to translate each field. Apply the resulting files with [Chronoctl](/tooling/chronoctl). ## Compare configurations These are examples of matching configurations for Datadog and Observability Platform. This is an example of a Datadog monitor definition. ```text theme={null} "id": 1234567, "org_id": 12345, "type": "metric alert", "name": "IOWAIT is high ({{value}})", "message": "{{#is_alert}}\Load is too high, check and lower load immediately (use AWS console for {{pod.name}} to scale tasks to 1 and investigate)\n@slack-ops-bots \n{{/is_alert}} \n\n{{#is_alert_recovery}}\n@slack-ops-bots \n@pagerduty-resolve \n{{/is_alert_recovery}}{{#is_warning}}\nLoad is reaching the limit.\n@slack-ops-warning-bots\n@pagerduty{{/is_warning}}", "tags": [ "high-load", "team:platform" ], "query": "min(last_30m):max:system.cpu.iowait{function:cassandraevents} by {pod,name} > 20", "options": { "notify_audit": false, "locked": false, "timeout_h": 0, "include_tags": true, "no_data_timeframe": 30, "require_full_window": true, "notify_by": ["pod"], "notify_no_data": true, "new_group_delay": 60, "renotify_interval": 30, "renotify_occurrences": 1, "renotify_statuses": [ "alert", "no data", ], "scheduling_options": { "evaluation_window": { "hour_starts": 30 } }, "thresholds": { "critical": 20, "critical_recovery": 10, "warning": 15 }, "timeout_h": 12, "escalation_message": "{{#is_alert}}\nEscalated to pagerduty - \nLoad is too high, check and lower load immediately (use AWS console for {{pod.name}} to scale tasks to 1 and investigate)\n@slack-ops-bots \n@pagerduty \ n{{/is_alert}}", "evaluation_delay": 300, "min_failure_duration": 120, "silenced": {} }, "multi": true, "created_at": 1479858941000, "created": "2016-11-22T15:55:41.80188-08:00", "modified": "2021-10-14T09:23:36.750186-07:00", "deleted": null, "restricted_roles": null, "priority": 1, "overall_state_modified": "2022-07-05T06:13:14-07:00", "overall_state": "OK", "creator": { "name": "Jane Smith", "handle": "janesmith@example.com", "email": "janesmith@example.com", "id": 18219 }, "matching_downtimes": [] ``` Observability Platform configuration breaks down into smaller, more strictly defined files covering the same functions. These file definitions use the `v1/config` resource format that you apply with [Chronoctl](/tooling/chronoctl). Configure the monitor: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Monitor spec: slug: cluster-iowait-high name: IOWAIT is high labels: datadog_id: "1234567" tag_iowait: "true" team: platform annotations: message_critical: Load is too high, check and lower load immediately (use AWS console for {{pod.name}} to scale tasks to 1 and investigate) message_warning: Load is reaching the limit prometheus_query: signal_grouping: label_names: - name series_conditions: defaults: critical: conditions: - op: GT value: 20 sustain_secs: 120 warn: conditions: - op: GT value: 15 sustain_secs: 120 notification_policy_slug: iowait-notification-policy interval_secs: 60 ``` This is the notification policy: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: NotificationPolicy spec: slug: iowait-notification-policy name: iowait notification policy routes: defaults: critical: notifier_slugs: - default-route-slack-bot warn: notifier_slugs: - default-route-slack-bot overrides: - alert_label_matchers: - name: tag_iowait type: EXACT value: "true" notifiers: critical: notifier_slugs: - slack-ops-bots - pagerduty-critical warn: notifier_slugs: - slack-ops-warning-bots - pagerduty-warning ``` And separate notifier routes: Slack, for an Alerts channel: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Notifier spec: name: Default route slack bot slug: default-route-slack-bot skip_resolved: false slack: api_url: channel: alerts username: oncall-bot ``` Slack, for an on-call channel: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Notifier spec: name: slack-ops-bots slug: slack-ops-bots skip_resolved: false slack: api_url: channel: oncall username: oncall-bot ``` Critical alerts sent to PagerDuty: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Notifier spec: name: pagerduty-critical slug: pagerduty-critical skip_resolved: false pagerduty: routing_key: url: https://events.pagerduty.com/v2/enqueue severity: critical ``` Warnings sent to PagerDuty: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Notifier spec: name: pagerduty-warning slug: pagerduty-warning skip_resolved: false pagerduty: routing_key: url: https://events.pagerduty.com/v2/enqueue severity: warning ``` ## Field mapping Chronosphere and Datadog fields have many equivalent functions. Use the following tables to map fields between these apps. Names of Chronosphere equivalents are subject to change as the conversion process improves. ### Configuration mapping This table matches Datadog fields to their Chronosphere equivalents for monitor specification. | Datadog field | Chronosphere equivalent | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `created` | N/A | | `creator` | N/A | | `id` | Add to `Monitor.labels`. | | `message` | Add to `Monitor.annotations` and create `Notifier` resources. See details. | | `modified` | N/A | | `multi` | `Monitor.spec.signal_grouping.signal_per_series` | | `name` | `Monitor.name` - This can also contain variables. | | `options` | [Monitor options](#monitor-options) | | `threshold_windows` | N/A - Used only for `anomalies`. | | `thresholds` | `Monitor.spec.series_conditions.defaults..conditions` | | `timeout_h` | N/A | | `overall_state` | For monitors with an `Ignored / Skipped / Unknown` state, still create the monitor but have it either go to a black hole route or create it as muted. | | `priority` | Can support as a message annotation. | | `query` | `Monitor.spec.prometheus_query` | | `restricted_roles` | N/A | | `state` | N/A | | `matching_downtimes` | Equivalent to schedules. | | `tags` | An arbitrary list of strings that fits the tag format (which can be single word tags). Chronosphere can support this using `Monitor.labels`, if the field requires a key-value format. Tags are used as label names with the value set to `true.` | | `type` | The type of monitor. Chronosphere supports query alert and metric alerts. | ### Monitor options Use these values in the specification's `options` field. | Datadog field | How to map | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `aggregation` | N/A - For log alerts only. | | `enable_logs_sample` | N/A - For log alerts only. | | `enable_samples` | N/A - Per [Datadog docs](https://docs.datadoghq.com/api/latest/monitors/#get-all-monitor-details). This is used only by CI Test and Pipeline monitors. | | `escalation_message` | No separate message for renotify notifications; can append this to the generic alert message. | | `evaluation_delay` | Can support by using offset in the query. | | `group_retention_duration` | N/A - Not for metrics monitors. | | `groupby_simple_monitor` | N/A - For log alerts only. | | `include_tags` | Use Prometheus `{{ $value }}` template. | | `min_failure_duration` | `Monitor.spec.series_conditions.defaults..conditions.sustain_secs` | | `min_location_failed` | Can support by adding thresholds to the PromQL expression. | | `new_group_delay` | N/A | | `new_host_delay` | N/A - Deprecated, use `new_group_delay` instead. | | `no_data_timeframe` | Threshold for a `no data` alert. See [severity](#severity) section for details. | | `notification_preset_name` | N/A - Datadog [docs](https://docs.datadoghq.com/api/latest/monitors/#get-all-monitor-details). | | `notify_audit` | N/A | | `notify_by` | Equivalent to `Monitor.spec.signal_grouping`, except the inverse. Note: This can be set to `*`, which is the same as setting `Monitor.spec.signal_grouping.signal_per_series`. | | `notify_no_data` | Add a `NOT_EXISTS` series condition in the MonitorSpec. Review [severity](#severity) for details. | | `on_missing_data` | N/A - Not for metrics alerts. | | `renotify_interval` | `NotificationPolicy.routes.overrides.notifiers..repeat_interval_secs` | | `renotify_occurrences` | N/A | | `renotify_statuses` | Only renotify on status X. Create overrides using `NotificationPolicy.routes.overrides.notifiers..repeat_interval_secs` for each severity listed here. | | `require_full_window` | Only evaluate if there's a full window of data. Datadog recommends setting this to `false`. Supportable using the `count_over_time` function. | | `scheduling_evaluation_window` | [Cumulative time windows](https://docs.datadoghq.com/monitors/configuration/?tab=thresholdalert#cumulative-time-windows). For example, "evaluate this alert every hour on the :00 mark." | | `silenced` | Dictionary of [muted tags to end timestamp](https://docs.datadoghq.com/monitors/guide/monitor_api_options/#common-options). Create MutingRule objects for each tag. | | `thresholds` | Thresholds for severity. Can map to `Monitor.spec.series_conditions.defaults` for warning and critical. No support for separate thresholds for recovery. | | `variables` | N/A | ## Severity Chronosphere supports both critical and warning severities by implementing different thresholds for the metric values. In addition to this, Datadog also supports alerting on no data for a particular metric as a distinct `severity`. Although this state isn't a true severity, the state is treated the same as critical and warning alerts for configuration. Chronosphere supports alerting on `no data` conditions using a series condition in the MonitorSpec: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Monitor spec: prometheus_query: series_conditions: defaults: critical: conditions: - op: NOT_EXISTS sustain_secs: 60 ``` ## Message and route Datadog allows different messages and routing endpoints for the different severity levels (critical, warning, no data). Chronosphere can support different messages by using separate annotations: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Monitor spec: annotations: message_critical: This is the critical threshold message message_warning: This is the warning threshold message message_no_data: This is the message for no data ``` To support different routes, users must use a separate monitor with different labels, set using notification policies. ## Notification policy resources Link a `Monitor` resource to a `Notifier` resource by defining a [notification policy](/investigate/alerts/notifications/policies). Each unique route in the Datadog message field maps to a `Notifier` resource. The `Monitor` contains a `label` specifying the notification route it links to, and the `NotificationPolicy` defines overrides that point to each `Notifier`. For example: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: Monitor spec: labels: datadog_id: "1234567" route_slack_ops_bots_critical: "true" route_slack_ops_bots_warning: "true" route_pagerduty_critical: "true" --- api_version: v1/config kind: NotificationPolicy spec: routes: overrides: - alert_label_matchers: - name: route_slack_ops_bots_critical type: EXACT value: "true" notifiers: critical: notifier_slugs: - slack-ops-bots ``` ## Evaluation frequency Datadog doesn't support the use of different evaluation frequencies per monitor, but instead relies on a hard-coded interval dependant on the [evaluation window](https://docs.datadoghq.com/monitors/configuration/?tab=thresholdalert#evaluation-frequency). For windows of less than `24h`, the window defaults to `1m`. Set this to a value you want to use with the `Monitor.spec.interval_secs` field (in seconds), or default to `15` to receive faster alerts. # Dashboard widgets Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/mappings/datadog/dashboards Map common Datadog dashboard widgets to Chronosphere Observability Platform standard dashboard panels. Some [Datadog](/ingest/metrics-traces/collector/mappings/datadog) dashboard widgets have a close match in Chronosphere Observability Platform [standard dashboards](/observe/dashboards). Recreate them with [panels](/observe/dashboards/panels) on a standard dashboard, not a [classic dashboard](/observe/dashboards/classic-dashboards). Before you rebuild visuals, translate Datadog queries to [PromQL](/investigate/querying/promql). For more information, see [Querying metrics](/ingest/metrics-traces/collector/mappings/datadog/dogstatsd), [Mapping](/ingest/metrics-traces/collector/mappings/datadog/mapping), and [Migration considerations](/ingest/metrics-traces/collector/mappings/datadog/migration-issues). ## Widget overview | Datadog widget | Observability Platform panel | Notes | | -------------------------- | ----------------------------------------------------------------- | -------------------------------------------------- | | Time series (stacked bars) | [Time series chart](/observe/dashboards/panels/time-series-chart) | Plots values across the dashboard time range | | Top list | [Bar chart](/observe/dashboards/panels/bar-chart) | Set **Query type** to **Instant** in **Options** | | Top list (tabular) | [Table](/observe/dashboards/panels/table) | Use when you need columns or multiple calculations | | Query value | [Stat chart](/observe/dashboards/panels/stat-chart) | Single aggregated value | Observability Platform doesn't directly import Datadog dashboard JSON. Rebuild panels manually or with your migration plan and tooling. ## Stacked bar graph Datadog time series widgets with stacked bars show values over time. Match them with a **Time series chart** panel. 1. [Create a panel](/observe/dashboards/panels#create-panels-in-a-dashboard) of type **Time series chart**, or [edit an existing panel](/observe/dashboards/panels#edit-panels-on-a-dashboard) and [change its type](/observe/dashboards/panels#change-a-panels-type-cloud) to **Time series chart**. 2. In the **Query** tab, enter a PromQL query for the series you want to stack. 3. In the **Settings** tab, in the **Visual** section: * Set **Display** to **Bar**. * Set **Stack series** to **All** for stacked bars, or to **Percent** for percent-stacked bars. 4. Optional: In the **Settings** tab, in the **Tooltip** section, set **Sort** to **Descending** instead of **Auto** or **None**. 5. In the **Edit panel** interface, click **Apply**. 6. On the dashboard, click **Save**. After you [convert the Datadog query](/ingest/metrics-traces/collector/mappings/datadog/dogstatsd), the following example shows one pattern for stacked values over time. ```text theme={null} sum by (LABEL_NAME) (rate(METRIC_NAME[5m])) ``` Replace the following: * *`METRIC_NAME`*: Your Prometheus metric name. * *`LABEL_NAME`*: The label you group by. ## Top list Datadog top list widgets rank series by a single value at a point in time. The closest match is a **Bar chart** panel. 1. [Create a panel](/observe/dashboards/panels#create-panels-in-a-dashboard) of type **Bar chart**, or [edit an existing panel](/observe/dashboards/panels#edit-panels-on-a-dashboard) and [change its type](/observe/dashboards/panels#change-a-panels-type-cloud) to **Bar chart**. 2. In the **Query** tab: * In **Options**, set **Query type** to **Instant**. * Enter a PromQL query with one value per series. 3. In the **Settings** tab, in the **Misc** section, set **Mode** to **Value** for absolute values, or to **Percentage** for values as a percent of the total. 4. Optional: In the **Settings** tab, in the **Misc** section: * Set **Sort** to **Descending (value)** if it isn't already. New bar chart panels default to descending sort. * Set **Limit** to cap how many bars display. If your query uses `topk`, set **Limit** to the same value as the `topk` argument, or omit `topk` from the query and use **Limit** only. 5. In the **Edit panel** interface, click **Apply**. 6. On the dashboard, click **Save**. Use an instant query that returns one value per series. The `topk` argument limits how many series the panel ranks, similar to a Datadog top list limit. The following example shows one pattern for a ranked instant query. ```text theme={null} topk(10, sum by (LABEL_NAME) (rate(METRIC_NAME[5m]))) ``` Replace the following: * *`METRIC_NAME`*: Your Prometheus metric name. * *`LABEL_NAME`*: The label you group by. ### Top list as a table Use a **Table** panel when you need label columns, multiple calculations, or a sortable grid instead of horizontal bars. 1. [Create a panel](/observe/dashboards/panels#create-panels-in-a-dashboard) of type **Table**, or [edit an existing panel](/observe/dashboards/panels#edit-panels-on-a-dashboard) and [change its type](/observe/dashboards/panels#change-a-panels-type-cloud) to **Table**. 2. In the **Query** tab: * In **Options**, set **Query type** to **Instant**. * Enter a PromQL query with one value per series. 3. In the **Settings** tab: * In the **Data Format** section, set **Time series format** to **Series To Rows**. * Under **Calculations**, select **Total**. * In the **Sorting** section, set **Sort by** to **Total** and **Sort order** to **Descending**. 4. Optional: In the **Settings** tab, in the **Data Format** section, enable **Label-based columns** to show each series's labels. 5. In the **Edit panel** interface, click **Apply**. 6. On the dashboard, click **Save**. Use an instant query that returns one value per series. The `topk` argument limits how many rows display, similar to a Datadog top list limit. The following example shows one pattern for a ranked instant query. ```text theme={null} topk(10, sum by (LABEL_NAME) (rate(METRIC_NAME[5m]))) ``` Replace the following: * *`METRIC_NAME`*: Your Prometheus metric name. * *`LABEL_NAME`*: The label you group by. # Querying DogStatsD formatted metrics Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/mappings/datadog/dogstatsd Translate Datadog query syntax into PromQL to query DogStatsD-formatted metrics in Chronosphere Observability Platform. Chronosphere can ingest and use Datadog metrics. The query syntax differs from PromQL, which requires you to build queries differently. ## Anatomy of a Datadog query The following example illustrates the structure of a Datadog query. Other queries might be in a different order: ```text theme={null} avg(last_1d):avg:count_nonzero(uptime{app:shopist} by {host}.as_rate()).rollup(avg,3600)<2 ``` This query breaks down into these sections: * Evaluation window: `avg(last_1d)` * Space aggregator: `avg` * Function: `count_nonzero` * Metric name: `uptime` * Filters or scope: `app:shopist` * Grouping: `host` * Type converter: `as_rate()` * Functions: `rollup(avg,3600)` * Operators: `<2` For more information and examples, see the Datadog documentation for [tracing](https://docs.datadoghq.com/tracing/trace_explorer/query_syntax) and [metrics query syntax](https://docs.datadoghq.com/metrics/#querying-metrics). ## Query syntax and modes Querying DogStatsD metrics in Chronosphere are based on the different modes [set in the Collector](/ingest/metrics-traces/collector/addl-metrics/dogstatsd#dogstatsd-ingestion). Metrics storage in the backend depends on the `mode` configured in the `dogstatsd` section of the `push` configuration in the Collector. The query syntax is slightly different for each `mode`. * `regular` The DogStatsD `METRIC_NAME` maps to the Prometheus `__name__` label, replacing all non-alphanumeric and non-dot characters with underscores. Dots convert to an underscore ( `_` ). Any labels defined on the metric remain unchanged and append to the list of labels. Refer to [Prometheus naming recommendations](/ingest/metrics-traces/collector/mappings/prometheus/prometheus-recommendations) for specific information. * `graphite` The Prometheus `__name__` label gets a constant `stat` name and the DogStatsD `METRIC_NAME` assigns to a Prometheus label set in the configuration `namelabelname` (by default `name`). * `graphite_expanded` The expanded Graphite mode is the same as `graphite` mode, except in addition to storing everything in the `namelabelname` label, the `METRIC_NAME` separates on dot ( `.` ) and stores each part in a separate label. For example, `t0`, `t1`, and `t2`. Here's an example of a DogStatsD query: ```text theme={null} users.online:2\|c\|#country:france ``` The following table shows examples of the same query in each of the Collector mode configurations: | Mode | Metric Output | | ------------------- | --------------------------------------------------------------------------------- | | `regular` | `users_online{country="france"}` or `{__name__="users_online", country="france"}` | | `graphite` | `stat{name="users.online", country="france"}` | | `graphite_expanded` | `stat{name="users.online", t0="users", t1="online", country="france"}` | ## Querying best practices For `graphite_expanded` metrics, it's best to start your query with `stat`, and then search for either `t0` or the defined labels using autocomplete. By starting with `stat`, your search scope focuses on the DogStatsD metrics, which improves query performance. For example, using the previous metric (`users.online:2|c|#country:france`), you can start your query with `stat`, add `t0` and using autocomplete, and search for `users`. Then, search for `t1` and so on. ## Metric types and querying All metrics convert to [Prometheus metric types](/control/shaping/shape-metrics/types) before storage in Chronosphere. Most metric types are the same across DogStatsD and Prometheus with the exception of counters. Counters in Prometheus are running counters, which means they always increase or remain constant, and never decrease. DogStatsD counters are `DELTA` counters. When querying counters in Chronosphere, apply a [`rate ()` function](https://prometheus.io/docs/prometheus/latest/querying/functions/#rate). ### Querying Prometheus counters In Prometheus, counters increase monotonically and must be wrapped in either a `rate` or `increase` function. Chronosphere conversion tooling attempts to fetch the metric type from Datadog. In this case of network issues or the metric not existing on Datadog side, it falls back to doing a substring match (ending in `_total`, `_count`, and so on). As an example, `gke_event_reception_client_track_event` doesn't end in a typical counter-like suffix so Chronosphere assumes that it's a gauge if the metric type fetch fails. The converted query might look like this: ```text theme={null} sum_over_time(sum(gke_event_reception_client_track_event{env="prod",event="sent"})[5m:]) ``` The corrected query should look like this: ```text theme={null} sum(rate(gke_event_reception_client_track_event{env="prod",event="sent"}[5m])) ``` You can tell at query time that a metric is a counter if the value climbs monotonically to the right. ### Convert cumulative histogram queries To correctly query histograms in Prometheus, you need to know the correct patterns. Unlike Datadog distributions, Prometheus histograms use the `_bucket` suffix. When doing a sum by condition, you must include `le`. #### Query for quantiles If your original Datadog query is: ```text theme={null} p75:prom.compression_request_time_milliseconds{} by {codec} ``` The correct Prom query will be: ```text theme={null} histogram_quantile(.75, sum by (le, codec)(rate(prom_compression_request_time_milliseconds_bucket{}[5m]))) ``` #### Query for average If your original Datadog query is this: ```text theme={null} avg:prom.compression_request_time_milliseconds{} by {codec} ``` The correct PromQL query will be: ```text theme={null} sum by (codec) (rate(prom_compression_request_time_milliseconds_sum{}[5m]))) / sum by (codec) (rate(prom_compression_request_time_milliseconds_count{}[5m]))) ``` The generic form is: ```text theme={null} sum(rate(foo_histogram_sum{}[5m]))/sum(rate(foo_histogram_count{}[5m])) ``` #### Min and max Convert histogram `min` and `max` by taking the `histogram_quantile(0, ...)` and `histogram_quantile(1, ...)` respectively. ### Convert exponential histogram queries It's important to know the pattern for correctly querying exponential histograms in Prometheus coming from querying for distributions in Datadog. #### Query for quantiles If your original Datadog query is this: ```text theme={null} p50:render_latency.latency{} ``` The correct PromQL query will be: ```text theme={null} histogram_quantile(.5, sum(rate(render_latency{}[5m]))) ``` #### Query for average Exponential histograms have some special functions to calculate `avg`, `min`, `max`, and `count`. These functions are `histogram_avg()`, and `histogram_count()`. If your original Datadog query is this: ```text theme={null} avg:render_latency{} by {codec} ``` The PromQL query will be: ```text theme={null} histogram_avg(sum by (codec) (rate(render_latency{}[5m]))) ``` #### Min and max Histogram `min` and `max` can be converted by taking the `histogram_quantile(0, ...)` and `histogram_quantile(1, ...)`, respectively. #### Advanced: Take the one-hour average of the p99 of a histogram Any PromQL query can be wrapped in any `_over_time()` function. To do so, you must leverage PromQL subquery syntax. The generic format is: `_over_time(()[1h:])`. Without the subquery syntax `[1h:]`, you will see an error like `parse error: ranges only allowed for vector selectors`. In PromQL, the `[1h:]` subquery syntax is necessary when wrapping a query with an `_over_time()` function because these functions operate on time series data over a range of time. The `[1h:]` specifies a time range (`1h`) for the subquery and a default resolution (`:`) for how often to evaluate the data points within that range. This creates a set of data points over the specified time range that the `_over_time()` function can process. If your original Datadog query is this: ```text theme={null} p99:prom.cloudtask_handler_time_ms{*}.rollup(avg, 3600) ``` The correct PromQL query will be: ```text theme={null} avg_over_time(histogram_quantile(.99, sum by(env, service_name) (rate(prom_cloudtask_handler_time_ms{}[5m])))[1h:]) ``` ## Query differences between Datadog and Chronosphere There are syntax differences between Chronosphere and Datadog queries. When you see differences in your data between the platforms, the following sections can help you determine the cause. ### Differences in interval If there are differences in the data being displayed in panels between Datadog and Chronosphere, review the time windows being used to see if they're different. Datadog can default to displaying a 30 minute time window for deltas, while Chronosphere defaults to 10 minutes. Adjust the query to use the same time window and `min` step interval to validate the data. The following images show examples of these differences: Datadog displaying 2-hour deltas for the past 7 days: Datadog displaying 2-hour deltas for the past 7 days Observability Platform displaying 10-min counter increases for the past 7 days (values are smaller): Observability Platform displaying 10-min counter increases for the past 7 days Same metric with a 2-hour counter increase for the past 7 days: Observability Platform displaying the same metric with a 2-hour counter increase for the past 7 days ### Set the Min step Prometheus, like Datadog, defaults to using a step size which is a function of the user interface's window size and query time window. Although you might want a line chart showing trends over time, a bar chart using sum the of values in the chart displays values higher than the actual values. Chronosphere recommends [setting the `Min step` option](/investigate/querying/metrics/explorer#define-a-querys-minimum-step-period) equal to the interval used in the query. In dashboards, you can use the `$interval` variable in both places. ### Handle label mismatch in division using `group_left` and `ignoring` Vector matching will fail when doing arithmetic on time series with different label sets. In this example, division fails when grouping by `label_A` and `label_B` in the numerator, but only `label_A` in the denominator. ```text theme={null} sum by (label_A, label_B) (metric) / sum by (label_A) (other_metric) ``` The pattern to correctly write this query is as follows: ```text theme={null} sum by (label_A, label_B) (metric) / ignoring(label_A) group_left() sum by (label_A) (other_metric) ``` ### Sum multiple sparse series Unlike Datadog, PromQL doesn't have behavior to `infer null as 0`. This means when you try summing together multiple sparse time series, the result will be null if any individual time series is null. For example, take the following query: ```text theme={null} sum(requests_succeeded{}) + sum(requests_failed{}) ``` If `requests.failed` only ever comes intermittently, the resulting addition would only produce a value when both `requests.succeeded` and `requests.failed` return values simultaneously. To solve this problem, Chronosphere recommends concatenating the metrics together on `__name_`: ```text theme={null} sum({__name__=~"requests_succeeded|requests_failed"}) ``` Following this pattern, Prometheus will essentially merge the time series together. ### Complex Boolean logic in filters Datadog has support for complex Boolean conditionals in label filters. Take the following query: ```text theme={null} sum:my.metric{NOT error:404 AND NOT (namespace:foo AND error:503)} ``` A simplistic approach to convert this query would result in ```text theme={null} sum(my_metric{error!="404", namespace!="my.namespace", error!="503"}) ``` However, this is incorrect. Taking a step back, the original Datadog query translates to: * `NOT error:404`: Select all metrics except those with error:404. * `AND`: Both conditions need to be satisfied. * `NOT (namespace:my.namespace AND error:503)`: Select all metrics except those with `namespace:my.namespace` and `error:503` together. To correctly convert this query to PromQL while preserving the logic, it should be: ```text theme={null} sum(my_metric{error!="404"} unless (my_metric{namespace="my.namespace", error="503"})) ``` Because `my_metric{error!="404"}` filters out metrics where error is 404 unless PromQL is used to exclude a subset of the data that matches certain labels from the main set. `my_metric{namespace="my.namespace", error="503"}` defines the subset to exclude, which is those with `namespace:my.namespace` and `error:503`. This conversion ensures the correct logical interpretation of the original Datadog query. # Datadog migration considerations Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/mappings/datadog/migration-issues Review questions about metrics, dashboards, and alerts to plan a migration from Datadog to Chronosphere Observability Platform. Datadog uses its own proprietary query language and storage system. Chronosphere is based on Prometheus. As you consider moving from Datadog to Chronosphere, review the following questions and suggestions as you plan your migration. ## General * Are you planning to stay on DogStatsD, or move to [Prometheus](https://prometheus.io/) or [Open Telemetry](https://opentelemetry.io/)? * How's your cluster defined? ### Metrics Understanding the metrics you're using and how they're used can help determine the time needed to switch from the previous system to Chronosphere. * Do you have a list of the metrics you are using for dashboards and alerts? * How many dashboards do you need to convert? * How many widgets per dashboard? * Determine the breakdown of widget type (visualization) per dashboard. * Which dashboards are critical to you and your users? You can use [popularity ranking in Datadog](https://docs.datadoghq.com/dashboards/#popularity) to determine this: * [Get All Dashboards](https://docs.datadoghq.com/api/latest/dashboards/#get-all-dashboards). * Iterate over these dashboards' IDs: [Get a dashboard](https://docs.datadoghq.com/api/latest/dashboards/#get-a-dashboard). * How many monitors do you need to convert? * How do you route alerts in Datadog today? * Dynamic routing based off of labels. * Static routing per monitor. * Which monitors are critical? Monitors/alerts that are going to PagerDuty are often the most critical. Example: [Get All Monitor Details](https://docs.datadoghq.com/api/latest/monitors/#get-all-monitor-details). ## Integrations Chronosphere integrates with other systems. * What integrations are you currently using? * What metrics do those integrations use, either ingested or exposed. * How do you use those metrics in your dashboards and alerts? This helps create the correct exporter configuration. For example, configuring [YACE](https://github.com/nerdswords/yet-another-cloudwatch-exporter) to do a sum statistics aggregation versus average gives different results. * Do you have a plan for how to ingest the OSS metrics? * Get consensus about using OSS Grafana dashboards for OSS metrics, or if you'll need to convert the widgets. ## Custom metrics You and your users might have created custom metrics for internal applications. * How many metrics are distributions? Egress costs might be higher for distribution metrics than for other types. * Develop an understanding of what your custom metrics are, what they're monitoring, and what labels they have. ## Alerts migration Alerts and notifications are often critical parts of your observability platform. * Obtain a list of the existing alerting platforms being used. * Create a list of the notification channels in use. # Differences between the Prometheus Operator and Chronosphere Collector Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/mappings/prometheus/operator-collector Compare the Prometheus Operator and Chronosphere Collector for deployment, alerting, service discovery, and long-term storage. ## Summary | Requirement | Prometheus Operator | Chronosphere Collector | | --------------------------------------- | ------------------------------------------------ | -------------------------------------------------- | | Deployment | StatefulSets managed by Prometheus Operator | Sidecar, Deployment, or DaemonSet | | Alerting | AlertManagerConfig, AlertManager, PrometheusRule | Alerts and Monitors | | High availability and long-term storage | Provided by Thanos | Natively supported | | Service discovery | Probe, PodMonitor, ServiceMonitors | Annotations, ServiceMonitors, Prometheus Discovery | ## Alerting Chronosphere Observability Platform supports Prometheus alerts, but not [AlertManagerConfig](https://prometheus.io/docs/alerting/latest/configuration/) or [AlertManager custom resource definitions (CRDs)](https://prometheus-operator.dev/docs/getting-started/design/#alertmanager). Observability Platform alerting (called *monitors*) has concepts that don't apply to Prometheus alerting rules, has its own concepts and models, and doesn't support complex routing trees. For more information, refer to the [monitors documentation](/investigate/alerts/monitors). Due to Observability Platform being a single data store, you can merge alerts, so an alert queries all metrics and not only metrics local to a Prometheus instance. You manage alerting configuration separate of any cluster or Chronosphere Collector configuration with Observability Platform, [Chronoctl](/tooling/chronoctl), or [Terraform](/tooling/infrastructure/terraform). This approach brings more flexibility for managing configuration and means you can spread configuration responsibility between teams. ## Scaling ### Thanos support Observability Platform is a scalable backend for Prometheus and doesn't require [Thanos](https://thanos.io) or the [ThanosRuler CRD](https://prometheus-operator.dev/docs/getting-started/design/#thanosruler). ### Sharding across instances The Prometheus Operator supports automatically sharding ServiceMonitors across multiple Prometheus instances. However, you still need to setup a remote write destination such as Thanos, or a single large instance. The Collector handles scale by using a DaemonSet and scoping each instance of the Collector to a particular node. Using a DaemonSet is the recommended way to deploy the Collector, but there are other methods available you can read about in the [Collector documentation](/ingest/metrics-traces/collector/install/kubernetes). There are advantages and disadvantages to deploying the Collector as a DaemonSet: * **Advantages**: * Using a DaemonSet means you don't need large or powerful instances to run the Collector. * The DaemonSet implementation reduces any impact of a single Collector instance experiencing issues. * **Disadvantages**: * All instances created with a DaemonSet must have uniform resources. ## Configure Prometheus Prometheus Operator has a [Prometheus CRD](https://prometheus-operator.dev/docs/getting-started/design/#prometheus) for configuring global settings on the instances it creates. Observability Platform supports many of these settings, but instead you set them in Collector configuration. For more information, visit the [configuration documentation](/ingest/metrics-traces/collector/configure). Some Prometheus Operator settings that don't transfer to the context of the Collector, such as `volumeMounts` and `priorityClass`. The Collector doesn't support the following fields from the ServiceMonitor CRD: * `targetLabels` * `podTargetLabels` Instead, use [Prometheus `relabel_config`](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) which allows advanced modifications to any target and its labels before ingesting the metrics. ## Service discovery ### Static scrape targets To replicate the capabilities of the [Prometheus Operator Probe CRD](https://prometheus-operator.dev/docs/getting-started/design/#probe), Chronosphere recommends running a single instance of the Collector as a sidecar (if possible), or a one instance Deployment. If you run the Collector as a DaemonSet, all instances of the Collector attempt to scrape the same targets, resulting in multiple copies of the same metrics. For more information about available options, refer to the [Collector documentation](/ingest/metrics-traces/collector/install/kubernetes#install-the-collector). # Prometheus metric naming recommendations Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/mappings/prometheus/prometheus-recommendations Follow Prometheus conventions for metric names and labels to keep telemetry consistent across teams in your organization. When choosing metric names and key-value pairs for labels, it's important to be consistent throughout your organization. Consistent labels reduce confusion across teams. A metric consists of a name and a set of labels, regardless of its [metric type](/control/shaping/shape-metrics/types). ```text theme={null} node_network_receive_bytes_total{hostname="host1",mode="system"} ``` The *name* of a metric is a single word that uniquely identifies the metric. In this example, the name is `node_network_receive_bytes_total`. A *label* is a key-value pairs that helps differentiate and give meaning to metric names. In this example, the labels are `hostname="host1"` and `mode="system"`. ## Metric name Use these guidelines to name your metrics: Metric names must follow [Prometheus name and label requirements](https://prometheus.io/docs/concepts/data_model/). Metrics that do not meet these requirements can not be queried and are considered invalid. Chronosphere rejects metrics that don't meet these definitions. * Select a metric name that describes a function of the system you're monitoring (for example, `node_memory_usage_bytes`). Be as concise as possible. * Include a prefix for the metric name that's relevant to the domain the metric belongs to. For example, adding `http` to `http_request_duration_seconds` lets users know this metric belongs to the HTTP domain. * Use suffixes to describe the unit in plural form. For example, the `seconds` in `http_request_duration_seconds`. * Metric names can contain letters, numbers, underscores, or colons, based on the regular expression `[a-zA-Z_:][a-zA-Z0-9_:]*`. * Reserve colons (`:`) in the name for calculated or aggregated metrics, such as those produced by rollup rules. * Don't use underscores at the beginning of a metric name. Leading underscores are reserved for internal use. * Periods (`.`) convert to underscores (`_`). ## Label names and values Labels are a set of comma-separated key-value pairs, contained in curly braces. Labels let you query and alert on these different fields. Use labels to differentiate and give meaning to metric names. For example, `http_requests_total` refers to the total number of HTTP requests, but isn't fully descriptive. Using labels, you can break down the number of requests by `method`, `status_code`, and `endpoint`. Here are some best practices and limitations: * Label keys * Use label keys to add dimensions to your metric name. For example, `endpoint:/api/create_user` or `method:POST`). * Don't add unbounded, high-cardinality labels, such as UUIDs or timestamps. * Label keys can contain letters, numbers, underscores, or colons, based on the regular expression `[a-zA-Z_:][a-zA-Z0-9_:]*`. * Don't use underscores at the beginning of a label key. Leading underscores are reserved for internal use. * Label values * Label values might contain any Unicode characters. * A label with an empty label value is equivalent to a label that doesn't exist. * You can use PromQL to filter and aggregate based on these dimensions. Changing any label value, including adding or removing a label, creates a new time series. # Migration notes for SignalFx (Splunk) to PromQL Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/mappings/signalfx Learn how to translate SignalFx queries to PromQL for Chronosphere Observability Platform. There are several key differences between SignalFx (Splunk) and PromQL of which to be aware when migrating queries to Chronosphere: * *Rollups* in SignalFx are used to aggregate data points within a single time series. With Chronosphere, rollups are [defined only on ingest](/control/shaping/shape-metrics/rules), and not at query time. However, PromQL offers a [set of functions](https://prometheus.io/docs/prometheus/latest/querying/functions/#aggregation_over_time) for aggregating individual time series over time. * *Analytics* in SignalFx are the equivalent to functions in PromQL. ## Equivalent function list | SignalFx | PromQL | Notes | | -------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------- | | Absolute value | `abs()` | *none* | | Bottom | `bottomk()` | *none* | | Ceiling | `ceil()` | *none* | | Count | `count()`, `count_over_time()` | *none* | | Delta | `delta()` | *none* | | EWMA and Double EWMA | `holt_winters()` | `holt_winters()` is similar to, but not exactly like, Exponentially Weighted Moving Average | | Exclude | Use comparators | *none* | | Floor | `floor()` | *none* | | Integrate | `rate()` | *none* | | LN - Log (natural) | `ln()` | *none* | | Log10 | `log10()` | *none* | | Maximum | `max()`, `max_over_time()` | *none* | | Mean | `avg()`, `avg_over_time()` | *none* | | Minimum | `min()`, `min_over_time()` | *none* | | Percentile | `quantile()`, `quantile_over_time()` | Use histogram\_quantile() with Histograms | | Power | Use operators | *none* | | Rate of change | `rate()` | *none* | | Scale | Use operators | *none* | | Square root | `sqrt()` | *none* | | Standard deviation | `stddev()`, `stddev_over_time()` | *none* | | Sum | `sum()`, `sum_over_time()` | *none* | | Timeshift | offset | *none* | | Top | `topk()` | *none* | | Variance | `stdvar()`, `stdvar_over_time()` | *none* | # Migration notes for Tanzu (Wavefront) to PromQL Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/mappings/wavefront Learn how to translate Tanzu Observability queries to PromQL for Chronosphere Observability Platform. WavefrontQL and PromQL are both used to visualize data, and some functions are very similar. Use this information for guidance as you convert queries from Wavefront to Prometheus. ## Functions | Wavefront | PromQL | Notes | | ------------------ | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `abs()` | `abs()` | *none* | | `exists()` | `absent()` | WavefrontQL must use the `ts()` function. | | Unsupported | `absent_over_time()` | *none* | | `ceil()` | `ceil()` | WavefrontQL must use the `ts()` function. | | Unsupported | `changes()` | *none* | | `limit()` | `clamp_max()` | *none* | | `floor()` | `clamp_min()` | *none* | | `day()` | `day_of_month()` | Wavefront needs to specify the time zone with the day function. PromQL returns results in Coordinated Universal Time (UTC). | | `weekday()` | `day_of_week()` | *none* | | `daysInMonth()` | `days_in_month()` | *none* | | `align()` | `_over_time()` | *none* | | `deriv()` | `deriv()` | *none* | | `exp()` | `exp()` | *none* | | Unsupported | `floor()` | *none* | | `percentile()` | `histogram_quantile()` | In PromQL, use `histogram_quantile(0.95, sum by (le) rate(my_histogram_metric_bucket[1m]))` to get the ninety-fifth percentile. | | `hw()` | `holt_winters()` | *none* | | `hour()` | `hour()` | *none* | | Unsupported | `idelta()` | Use only with gauges in PromQL. | | `ratediff()` | `increase()` or `irate()` | WavefrontQl `ratediff()` gives you the only difference in value between each two data points. | | `join()` | `label_join()` | Wavefront can do specific joins like SQL. | | `taggify()` | `label_replace()` | *none* | | `log()` | `ln()` | *none* | | Unsupported | `log2()` | *none* | | `log10()` | `log10()` | *none* | | `minute()` | `minute()` | PromQL returns results in Coordinated Universal Time (UTC). You must specify a time zone in WavefrontQL. | | `month()` | `month()` | PromQL returns results in Coordinated Universal Time (UTC). You must specify a time zone in WavefrontQL. | | `linearforecast()` | `predict_linear()` | WavefrontQL can specify a length of time in the past for use with predicting the linear regression. PromQL predicts using the entire series, for use only with gauges. | | `rate()` | `rate()` | *none* | | `flapping()` | `resets()` | *none* | | `round()` | `round()` | PromQL rounds up the tied values. | | Unsupported | `scalar()` | *none* | | Unsupported | `sort()` | PromQL `sort()` and `sort_desc()` show the data order in the Console view. Because Wavefront visualizes queries in charts instead of a console, they don't support this option. | | Unsupported | `sort_desc()` | PromQL `sort()` and `sort_desc()` show the data order in the Console view. Because Wavefront visualizes queries in charts instead of a console, they don't support this option. | | `sqrt()` | `sqrt()` | *none* | | `time()` | `time()` | *none* | | `timestamp()` | `timestamp()` | *none* | | Unsupported | `vector()` | *none* | | `year()` | `year()` | PromQL returns results in Coordinated Universal Time (UTC). You must specify a time zone in WavefrontQL. | ## Time functions | Wavefront | PromQL | Notes | | -------------- | ---------------------- | -------------------------------------------------------------------------- | | `mavg()` | `avg_over_time()` | *none* | | `min()` | `min_over_time()` | WavefrontQL must use the `ts()` function to match `min_over_time()`. | | `max()` | `max_over_time()` | WavefrontQL must use the following form to match `max_over_time()`. | | `msum()` | `sum_over_time()` | `sum_over_time()` is applicable only to gauges. | | `mcount()` | `count_over_time()` | *none* | | `percentile()` | `quantile_over_time()` | WavefrontQL requires the rawpercentile() function instead of percentile(). | | | `stddev_over_time()` | *none* | | | `stdvar_over_time()` | *none* | ## Aggregations | Wavefront | PromQL | Notes | | ------------ | ---------------- | -------------------------------------------------------------------------------------------- | | `rawsum()` | `sum()` | *none* | | `min()` | `min()` | To emulate PromQl, use the following WavefrontQL form. | | `max()` | `max()` | To emulate PromQl, use the following WavefrontQL form. | | `avg()` | `avg()` | To emulate PromQL, use the following WavefrontQL form. | | | `group()` | *none* | | `stddev()` | `stddev()` | WavefrontQL can do only standard deviation on histograms. | | `variance()` | `stdvar()` | The `variance()` function uses interpolation. To not use interpolation, use `rawvariance()`. | | `count()` | `count()` | WavefrontQL must use the `ts()` function to approximate the effect of the PromQL function. | | `count()` | `count_values()` | To emulate PromQL function, use a Wavefront `events` function. | | `bottomk()` | `bottomk()` | *none* | | `topk()` | `topk()` | *none* | | | `quantile()` | *none* | ## Binary operators | Wavefront | PromQL | Notes | | --------- | ------ | ------ | | `.eq()` | `==` | *none* | | `.ne()` | `!=` | *none* | | `.gt()` | `>` | *none* | | `.lt()` | `<` | *none* | | `.ge()` | `>=` | *none* | | `.le()` | `<=` | *none* | ## Boolean operations | Wavefront | PromQL | Notes | | -------------- | -------- | ------------ | | `intersect()` | `and` | intersection | | `union()` | `or` | union | | `complement()` | `unless` | complement | # Verify and monitor the Collector Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/monitor Verify that the Chronosphere Collector is scraping metrics and receiving traces, and monitor Collector health in your clusters. To ensure the Collector is ingesting data, you can verify the Collector is [scraping the intended metrics](/ingest/metrics-traces/collector/monitor/verify-metrics) or [receiving trace data](/ingest/metrics-traces/collector/monitor/verify-traces) from your app. You can also [monitor the Collector](/ingest/metrics-traces/collector/monitor/monitor-collector) to ensure that Chronosphere receives telemetry data from your instances. If the Collector can't start or scrape metrics, you can [troubleshoot metrics ingestion](/ingest/metrics-traces/collector/monitor/monitor-collector#troubleshoot-metric-ingestion). # Monitor the Collector Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/monitor/monitor-collector Create monitors on kube-state-metrics to track Chronosphere Collector health and troubleshoot metric ingestion failures. If Collectors are unreachable, the self-scraping metrics might not reach Chronosphere. In these cases, you can [create a monitor](/investigate/alerts/monitors#create-a-monitor) for the `kube-state-metrics` key to examine the health of a Collector. This method of monitoring the Collector assumes that you're scraping `kube-state-metrics`, and that your Collectors typically report as expected. The following examples assume that Collector pod names include the `chronocollector` prefix. ## DaemonSet Collector is unavailable You can create a monitor for the following `kube-state-metrics` to determine whether a Collector running as a DaemonSet is unavailable. ### Pods are unavailable Monitor the following metrics to help determine whether DaemonSet pods are unavailable. ```text theme={null} (kube_daemonset_status_number_unavailable{daemonset="chronocollector"} * on (instance, host_ip, cluster) group_left(pod) kube_pod_info{pod=~"chronocollector.+", pod_ip!="", created_by_kind="DaemonSet"}) > 0 (kube_daemonset_status_desired_number_scheduled{daemonset="chronocollector"} * on (instance, host_ip, cluster) group_left(pod) kube_pod_info{pod=~"chronocollector.+", pod_ip!="", created_by_kind="DaemonSet"} - on (pod) kube_daemonset_status_number_available{daemonset="chronocollector"} * on (instance, host_ip, cluster) group_left(pod) kube_pod_info{pod=~"chronocollector.+", pod_ip!="", created_by_kind="DaemonSet"}) > 0 ``` ### Pod is restarting Monitor the following metrics to help determine if a `kube-state-metrics` Collector pod is restarting. The following query reviews the past five minutes to return any pod restarts for the `kube_pod_container_status_restarts_total` metric: ```text theme={null} rate(kube_pod_container_status_restarts_total{pod=~"chronocollector.+"}[5m]) > 0 ``` ### Container is restarting A container might restart for various reasons. For example, the Collector containers could be stuck in a crash loop due to a failed deployment, or there might be underlying platform issues. You can examine the service container logs to help determine what caused a container to restart, and also review the `kube_pod_container_status_terminated_reason` metrics. Create a monitor with the following query to return the rate of increase of the `kube_pod_container_status_restarts_total` metric over the past 15 minutes. This metric is helpful for detecting container restarts: ```text theme={null} increase(kube_pod_container_status_restarts_total{container=~"chronocollector"}[15m]) ``` ### Drops in the number of scraped targets A significant drop in the number of scrape targets can indicate issues with the Collector, or with the metric endpoints themselves. Create a monitor with the following query to calculate the percentage of active targets the Collector is scraping for a given instance. The query divides the rate of change in the number of active targets over the past hour by the maximum number of active targets over the past 24 hours: ```text theme={null} sum by (instance) (delta(chronocollector_k8s_gatherer_sink_targets_active{job=~".*chronocollector.*"}[1h] offset 1m))/ sum by (instance) (max_over_time(chronocollector_k8s_gatherer_sink_targets_active{job=~".*chronocollector.*"}[1h] offset 24h)) * 100 ``` ### Calculate scrape latency Create a monitor with the following query to calculate scrape latency. The query calculates the P99 latency for scrapes of `chronocollector_scrape_latency_bucket` metric values grouped by `job` and `le` labels: ```text theme={null} histogram_quantile(0.99, sum by(job, le) (rate(chronocollector_scrape_latency_bucket{job=~".*chronocollector.*"}[1m]))) ``` ### Detect error rates Create a monitor with the following query to detect Collector error rates. The query calculates the error rate of the Collector within a one-minute period: ```text theme={null} 1 - (sum by (instance) (rate(chronocollector_gateway_push_errors{job=~".*chronocollector.*"}[1m])))/(sum by (instance) (rate(chronocollector_gateway_push_errors{job=~".*chronocollector.*"}[1m])) + sum by (instance) (rate(chronocollector_gateway_push_success{job=~".*chronocollector.*"}[1m]))) ``` The output is a value between `0` and `1` that represents the error rate as a percentage of the total rate for the `chronocollector` job over the specified period. A value closer to `0` indicates a lower error rate, while a value closer to `1` indicates a higher error rate. ## Deployment Collector is unavailable When Chronosphere Collector actively scrapes metrics, it continuously generates an `up` metric for each target. This metric serves as a heartbeat for the Collector; if the `up` metric is unavailable, the Collector can't scrape `kube-state-metrics`, which means `kube-state-metrics` can't indicate the Collector's health. To track the `up` metric, [create a monitor](/investigate/alerts/monitors#create-a-monitor) that generates a notification if this metric is unavailable. ```shell theme={null} up{instance=~"chronocollector-ksm"} > 0 ``` ## Track network egress To assess the effects of compression and egress configuration changes, you can monitor the egress traffic from the Chronosphere Collector to Observability Platform's ingestion endpoints. To track the egress traffic in gigabytes over a 24-hour period, use the following query: ```shell theme={null} sum(increase(chronocollector_gateway_push_bytes{}[1d])) / (1024^3) ``` ## Troubleshoot metric ingestion If the Chronosphere Collector can't start or scrape metrics, use the following methods to troubleshoot issues. These steps assume that you're running the Collector in a Kubernetes environment. 1. Review the Collector logs for errors. Run the following command to view the Collector logs: ```shell theme={null} kubectl logs ds/chronocollector ``` For example, the following error indicates the scrape timeout is too low for the job default of `10s`: ```text theme={null} {"level":"info","ts":1578154502.096278,"msg":"","level":"debug","scrape_pool":"collector","target":"http://0.0.0.0:9100/metrics","msg":"Scrape failed","err":"Get http://0.0.0.0:9100/metrics: context deadline exceeded"} ``` 2. If the logs don't provide any insights, ensure the API token and gateway address exist as a secret in the Kubernetes cluster: ```shell theme={null} kubectl get secrets ``` 3. If there's no output to your terminal, a Secret doesn't exist. Create a Secret using `kubectl`: ```shell theme={null} kubectl create secret generic chronosphere-secret \ --from-literal=api-token=API_TOKEN \ --from-literal=address=TENANT:443 ``` Replace the following: * * 4. Ensure annotations are properly configured in each of the Pods you want to scrape. Refer to [Collector Kubernetes annotations](/ingest/metrics-traces/collector/discover#kubernetes-annotations) for more details. # Verify the Collector is scraping metrics Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/monitor/verify-metrics Verify that the Chronosphere Collector is scraping the metrics you expect by using Live Telemetry Analyzer and Collector statistics. Use the following methods to ensure the Collector is scraping the intended metrics. Contact [Chronosphere Support](/support) if there are other Collector statistics you want to access. Similarly, contact Chronosphere Support to request having specific telemetry data removed from Chronosphere Observability Platform. ## Live Telemetry Analyzer [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer) lets you inspect, in real time, the stream of metrics that Observability Platform is ingesting. You can use the Metrics Profiler to verify the Collector's connection to Observability Platform by viewing the metrics that the Collector emits about itself. 1. In the navigation menu, click ** Go to Admin** and then select ** Analyzers > Live Telemetry**. 2. Click the **Metrics** tab. 3. Click **Capture live data** to display streaming metrics. 4. In the **Keys** list, click the **`__name__`** and **instance** label keys. 5. In the **Values** filter, enter the following key:value pairs: ```text theme={null} __name__:chronocollector_* instance:INSTANCE_NAME ``` Replace *`INSTANCE_NAME`* with the host and port where the Collector is running. For example, a Collector running locally uses an instance name of `localhost:3030`. In the **Values** list, the displayed metrics include your Collector instance in the **instance** column. ## Metrics Explorer [Metrics Explorer](/investigate/querying/metrics/explorer) lets you validate metrics if you know the name of the metric or label you're searching for. 1. In the navigation menu select ** Explorers > Metrics Explorer**. 2. Enter the following query in the query field. ```text theme={null} count(chronocollector_jobs) by (instance) ``` 3. Click **Run**. The name of your Collector instance returned from the `kubectl logs` command displays in the table of metrics: | Time | Instance | | ------------------- | --------------------------------- | | 2023-02-03 10:56:04 | default/chronocollector-POD\_NAME | *`POD_NAME`* is the name of the Kubernetes pod where your Collector instance is running. ### Dashboards Observability Platform includes a **Collectors** dashboard by default that's actively maintained and updated. This [dashboard](/observe/dashboards) displays information about the metrics the Collector scrapes. When the Collector begins receiving metrics, the Collector dashboard panels populate and display statistics such as: * Number of Collectors running on the cluster * Number of metrics scraped per second * Number of scrape targets per job * Memory and CPU consumption * Push latency to Chronosphere Creating monitors based on some of these key metrics can help you detect if a Collector is performing poorly. Refer to the [default dashboard metrics](/overview/concepts/dictionary#collectors) for the full list of available metrics. In the navigation menu select **Dashboards** to access the Collector dashboard. * If you're using Kubernetes discovery, the Collector configuration includes an annotation for the Collector dashboard by default. * If you're using Prometheus discovery and don't see the Collector dashboard, ensure the following structure exists in your Collector configuration file under the `discovery` section: ```yaml theme={null} discovery: prometheus: - job_name: 'Collector' scrape_interval: 15s scrape_timeout: 30s static_configs: - targets: ['0.0.0.0:3030'] ``` # Verify the Collector is receiving traces Source: https://docs.chronosphere.io/ingest/metrics-traces/collector/monitor/verify-traces Verify that the Chronosphere Collector is receiving trace data by using Trace Analyzer and Trace Explorer. Use the following methods to ensure the Chronosphere Collector is receiving trace data. ## Trace Analyzer [Trace Analyzer](/investigate/analyze/trace-analyzer) provides a real-time view of incoming traces grouped by tag and their relative frequency. Use Trace Analyzer to display the stream of incoming spans from your trace data. 1. In the navigation menu, click ** Go to Admin** and then select ** Analyzers > Live Telemetry**. 2. Click the **Traces** tab. 3. Click **Capture live data** to display the stream of incoming spans. The default grouping shows spans by service (`__service__`), but you can include additional tags to group by such as `__trace_id__` and `__span_id__`. ## Trace Explorer [Trace Explorer](/investigate/querying/traces) helps you search for traces and spans so that you can identify, triage, and understand the root cause of problems. 1. In the navigation menu select ** Explorers > Trace Explorer**. 2. Modify the **Time Window** to the period when the Collector started ingesting trace data. The timeline graph displays a spike at the time when trace data started flowing into Chronosphere Observability Platform. # Ingest Google Cloud metrics Source: https://docs.chronosphere.io/ingest/metrics-traces/gcp Learn how to ingest Google Cloud metrics into Chronosphere Observability Platform. [Google Cloud](https://cloud.google.com/products/operations?hl=en) integration connects Chronosphere Observability Platform with Google Cloud to ingest metrics from Google Cloud projects. Observability Platform uses Google's [service account impersonation](https://cloud.google.com/iam/docs/service-account-impersonation) as the authentication mechanism to connect Google Cloud with Observability Platform. Service account impersonation is trust-based, and uses short-lived credentials so that one Google Cloud service account principal can impersonate another. Use the [Google Cloud Integration Dashboard](/observe/dashboards/managed-dashboards#google-cloud-integration-gcp) to review the status of your integration. If you want to route log data from Google Cloud Platform, see [Route logs from Google Cloud Platform](/ingest/logs/gcp-logs). ## Set up Google Cloud integration Successful ingestion of Google Cloud metrics requires infrastructure changes for your tenant, performed by Chronosphere, and changes to your Google Cloud, performed by you. You can perform your Google Cloud setup while Chronosphere works to provision your tenant with the updated infrastructure. 1. Contact [Chronosphere Support](/support) to request infrastructure changes to your tenant for Google Cloud integration. Follow the remaining steps to set up your Google Cloud project. Metrics won't flow into Chronosphere until infrastructure is provisioned for your tenant. 2. Modify the Google Cloud domain restriction constraint. If your Google Cloud organization restricts identities by [domain](https://cloud.google.com/resource-manager/docs/organization-policy/restricting-domains), you must add the Observability Platform customer identity (`C04aozwp9`) as an allowed value in your policy. 3. Create a Google Cloud service account. A Google Cloud service account isn't an Observability Platform service account. Observability Platform requires a Google Cloud service account in the user's Google Cloud project to provide Observability Platform access to metrics using service account impersonation. For each Google Cloud project, [create a service account](https://cloud.google.com/iam/docs/service-accounts-create) with the `monitoring.viewer` role. To instead grant a more restricted set of permissions, implementing the principle of least privilege, create a custom role with permissions for: * `monitoring.timeSeries.list` * `monitoring.metricDescriptors.list` * `monitoring.metricDescriptors.get` * `monitoring.monitoredResourceDescriptors.list` * `monitoring.monitoredResourceDescriptors.get` Chronosphere recommends [setting your project's quota](https://cloud.google.com/resource-manager/docs/creating-managing-projects#managing_project_quotas) to twice your normal usage to protect against spikes and unexpected usage. Use the [**GCP Timeseries API Quota Usage Per Project**](/observe/dashboards/managed-dashboards#gcp-timeseries-api-quota-usage-per-project) dashboard to monitor project usage. 4. Add the Observability Platform principal to the Google Cloud service account. Each Google Cloud service account must grant access to the Observability Platform principal to impersonate them. To retrieve your tenant-specific principal, click your [profile icon](/navigate#your-account) from the menu bar and select **My Account**. Copy the value displayed next to **Google Cloud Metrics IAM Principal**. Grant the principal the `iam.serviceAccountTokenCreator` role. 5. Configure the integration in Observability Platform. Initialize the integration using the [Google Cloud integration API](/tooling/api-info/definition/operations/ListGcpMetricsIntegrations) by applying the configuration with [Chronoctl](/tooling/chronoctl) or [Terraform](/tooling/infrastructure/terraform). For each Google Cloud project, supply the Google Cloud service account email (`SERVICE-ACCOUNT@PROJECT-ID.iam.gserviceaccount.com`) in the request to configure the integration, along with specific metrics to ingest using metric prefixes. See [Metric Prefix Search](#metric-prefix-search) for examples. ### Example configuration The following code provides an example for creating a single Google Cloud service account in the user's Google Cloud project, and enables Observability Platform to impersonate and gain access. ```terraform expandable Terraform example icon="square-terminal" theme={null} locals { // Email address of Observability Platform tenant-specific principal. chronosphere_sa_email = "SERVICE-ACCOUNT@PROJECT-ID.iam.gserviceaccount.com" // Google Cloud project containing monitoring data to be ingested into // Observability Platform. monitoring_project_id = "PROJECT_ID" } // Service account that will allow Observability Platform tenant-specific principal // to impersonate it. resource "google_service_account" "chronosphere" { project = local.monitoring_project_id account_id = "chronosphere" } // Custom role with minimal set of permissions for principle of least privilege // resource "google_project_iam_custom_role" "chronosphere" { project = local.monitoring_project_id role_id = "Google Cloud Integration Metrics Viewer" title = "Google Cloud Integration Metrics Viewer" description = "Role for granting view access for Google Cloud integration" permissions = [ "monitoring.metricDescriptors.list", "monitoring.timeSeries.list", ] } // The service account has the minimal set of permissions for access to the project // containing the metric data so that it can retrieve time series data from the // Google Cloud API. resource "google_project_iam_member" "chronosphere" { project = local.monitoring_project_id role = google_project_iam_custom_role.chronosphere.id member = google_service_account.chronosphere.member } // The service account provides the Observability Platform tenant-specific principal with // roles/iam.serviceAccountTokenCreator access so that it can impersonate it. Only // the Observability Platform tenant-specific principal will be able to perform this // impersonation. data "google_iam_policy" "chronosphere" { binding { role = "roles/iam.serviceAccountTokenCreator" members = ["serviceAccount:${local.chronosphere_sa_email}"] } } // Assign the token creator permission to the service account. resource "google_service_account_iam_policy" "chronosphere" { service_account_id = google_service_account.chronosphere.name policy_data = data.google_iam_policy.chronosphere.policy_data } ``` ### Use metrics scopes for projects Chronosphere encourages users to use [metrics scoping for projects](https://cloud.google.com/monitoring/settings/multiple-projects) in their Google Cloud environment, particularly for users that manage five or more Google Cloud projects. Chronosphere recommends creating a new metrics scoped project for use with metrics ingestion. Using a new scoped project minimizes the impact on your existing workload. Metrics scopes have benefits over configuring individual Google Cloud projects for Google Cloud integration: * Google Cloud API cost savings: With metrics scoping, Observability Platform ingests metrics using the least number of API requests to Google Cloud. Metrics scoped projects introduce a fan-in and fan-out model of ingesting metrics from the underlying Google Cloud projects that will pack time series data points for each metric into the smallest set of API responses. The Google Cloud API cost savings are significant, particularly for users managing multiple Google Cloud projects. * Configuration simplicity: With metrics scoping, users can manage a smaller set of Google Cloud service accounts and configuration in Observability Platform. This simplifies management of the Google Cloud integration. ## Set up Observability Platform to receive Google Cloud data After configuring Google Cloud to enable Observability Platform to access metrics, you must configure Observability Platform to receive and process those metrics. ### View Google Cloud integrations To list or view Google Cloud integrations, use one of the following options: To list your Google Cloud integrations: ```shell theme={null} chronoctl gcp-metrics-integrations list ``` To view a Google Cloud integration with Chronoctl, use the command: ```shell theme={null} chronoctl gcp-metrics-integrations read SLUG ``` Replace *`SLUG`* with the unique identifier of the Google Cloud integration. To list Google Cloud integrations with the Chronosphere API, use the [`ListGcpMetricsIntegrations`](/tooling/api-info/definition/operations/ListGcpMetricsIntegrations) endpoint. To view a single Google Cloud integration with the Chronosphere API, use the [`ReadGcpMetricsIntegration`](/tooling/api-info/definition/operations/ReadGcpMetricsIntegration) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Create or update a Google Cloud integration You can create or update your Google Cloud integration with Observability Platform by applying a configuration file with Chronoctl or Terraform. You must add your service account to an Observability Platform team with [SysAdmin](/administer/accounts-teams/teams#add-a-role-to-a-team) permissions. To create a Google Cloud integration with Chronoctl, use the command: ```shell theme={null} chronoctl gcp-metrics-integrations create --filename FILENAME ``` Replace *`FILENAME`* with the name of your Chronoctl configuration file. To update a Google Cloud integration with Chronoctl, use the command: ```shell theme={null} chronoctl gcp-metrics-integrations update --filename FILENAME ``` Replace *`FILENAME`* with the name of your Chronoctl configuration file. The input file uses the following structure: ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: GcpMetricsIntegration spec: name: NAME slug: SLUG service_account: client_email: EMAIL metric_groups: - project_id: PROJECT_ID prefixes: - PREFIX ``` To create a Google Cloud integration with Terraform, use the `chronosphere_gcp_metrics_integration` resource: ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_gcp_metrics_integration" "chronosphere_gcp_metrics" { name = "GCP Metrics" slug = "gcp-metrics" service_account { client_email = "metric-scoped-service-account@.iam.gserviceaccount.com" } metric_groups { project_id = "PROJECT_ID" prefixes = [ "PREFIX", ] } } ``` To create Google Cloud integrations with the Observability Platform API, use the [`CreateGcpMetricsIntegration`](/tooling/api-info/definition/operations/CreateGcpMetricsIntegration) endpoint. To update a single Google Cloud integration with the Observability Platform API, use the [`UpdateGcpMetricsIntegration`](/tooling/api-info/definition/operations/UpdateGcpMetricsIntegration) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. Replace the following: * *`NAME`*: (string) The name of the Google Cloud integration. * *`SLUG`*: (string) The unique identifier of the Google Cloud integration. * *`EMAIL`*: (string) The email address of the customer's Google Cloud service account of the form `SERVICE_ACCOUNT@PROJECT-ID.iam.gserviceaccount.com`. * *`PROJECT_ID`*: (string) The ID of the customer's Google Cloud project to ingest metrics from. * *`PREFIX`*: (list(string)) A list of [metric prefixes](#metric-prefix-search) to ingest from the Google Cloud project. #### Control incoming metrics Google Cloud charges users for metrics leaving their system, potentially incurring costs for scraped metrics that aren't needed. During ingestion into Observability Platform, users can apply [rollup](/control/shaping/shape-metrics/rules/rollup) and [drop](/control/shaping/shape-metrics/rules/drop-rules) rules to reduce ingestion and storage costs, but these apply after extraction from Google Cloud. To reduce the number of metrics extracted from Google Cloud, Observability Platform provides the following configuration options: * *Filters*: [Filters in Google Cloud](https://docs.cloud.google.com/monitoring/api/v3/filters) let you target a subset of metrics. Filters don't replace [prefixes](/ingest/metrics-traces/gcp#metric-prefix-search). Instead, filters apply to metric and resource labels, or other user-defined metadata. Due to constraints in the Google Cloud monitoring API, when filtering by user-defined metadata, the resulting metric type changes to `DELTA` in the case of a `CUMULATIVE` metric. Metric query expressions might need adjustment to use delta-specific query functions (for example, `sum_over_second`). * *Rollup Rules*: Rollup rules enable pre-aggregation of a metric by specifying which result dimensions should be returned. All other labels will be dropped. Use these rules to [reduce cardinality](/control/shaping/shape-metrics/reduce-cardinality) of returned time series from Google Cloud by dropping dimensions that aren't needed. Exclude all metrics where the `metadata.system_labels` "region" label matches `us-central1-a`: ```yaml theme={null} metric_groups: - project_id: PROJECT_ID prefixes: - PREFIX filters: - name: "region" value_glob: "!us-central1-a" context: "METADATA_SYSTEM_LABELS" ``` Selectively aggregate a subset of metrics for non-production environments, using a `metadata.user_labels` label of `cluster`. ```yaml theme={null} metric_groups: - project_id: PROJECT_ID prefixes: - PREFIX filters: - name: cluster value_glob: prod-* context: "METADATA_USER_LABELS" - project_id: PROJECT_ID prefixes: - PREFIX filters: - name: cluster value_glob: rnd-* context: "METADATA_USER_LABELS" rollup_rules: - metric_name: METRIC_NAME label_policy: keep: - name: cluster context: "METADATA_USER_LABELS" ``` ### Delete a Google Cloud integration Delete a Google Cloud integration using one of the following methods: To delete a Google Cloud integration with Chronoctl, your account must have [SysAdmin](/administer/accounts-teams/teams#add-a-role-to-a-team) permissions. Use the command: ```shell theme={null} chronoctl gcp-metrics-integrations delete SLUG ``` To complete this action with the Chronosphere API, use the [`DeleteGcpMetricsIntegration`](/tooling/api-info/definition/operations/DeleteGcpMetricsIntegration) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ## Metric information The following sections contain information about Google Cloud metrics information and how Observability Platform uses or displays it. ### Metrics availability Google metrics have different [stages of availability](https://cloud.google.com/document-ai/docs/reference/rest/Shared.Types/LaunchStage). Observability Platform prioritizes metrics Google considers Generally Available (GA). Non-GA metrics ingest based on a best-effort basis, and might not provide stable data. On average, Google Cloud metrics display in Observability Platform with a delay on the order of minutes from the metric timestamp in Google Cloud. These metrics appear delayed due to the following factors: * Latency delay: Most Google Cloud metrics have a [delay](https://cloud.google.com/monitoring/api/metrics#metadata) before they're made available for querying by Google Cloud API. Metrics with an ingest delay exceeding seven hours won't be ingested into Chronosphere. For example, Firebase database metrics can exceed this latency delay and won't be ingested. * Sample rate: Google Cloud metrics [sample](https://cloud.google.com/monitoring/api/metrics#metadata) at different rates. Most metrics sample at 60-second intervals, with some as long as once per day. Due to these delays, configure monitors to expect a delay in arrival. Use the PromQL [offset modifier](https://prometheus.io/docs/prometheus/latest/querying/basics/#offset-modifier) to account for delays. Observability Platform doesn't ingest the following networking flow metric prefixes due to their unpredictable data volumes and cardinality. If you have a specific need for metrics using these prefixes, contact [Chronosphere Support](/support). * `networking.googleapis.com/pod_flow` * `networking.googleapis.com/vm_flow` * `networking.googleapis.com/node_flow` ### Metric names In Observability Platform, [Google Cloud metrics](https://cloud.google.com/monitoring/api/metrics) have the following name structure: ```text theme={null} gcp__ ``` where: * `MONITORED-RESOURCE-NAME` is the [monitored resource type](https://cloud.google.com/monitoring/api/resources#tag_gce_disk) in Google Cloud. * `FULLY-QUALIFIED-METRIC-NAME` is the metric name in Google Cloud, with characters like `.` and `/` convert to `_` to be [Prometheus compatible](/ingest/metrics-traces/collector/mappings/prometheus/prometheus-recommendations). The following examples are fully qualified metric names in Google Cloud before metric name sanitization: * `bigquery.googleapis.com/job/num_in_flight` * `cloudsql.googleapis.com/database/available_for_failover` * `kubernetes.io/container/accelerator/memory_total` For example, to map [the CloudSQL metric `database/active_directory/domain_reachable`](https://docs.cloud.google.com/sql/docs/mysql/admin-api/metrics) to a metric name in Observability Platform, `cloudsql_database` is the monitored resource name and `cloudsql.googleapis.com/database/active_directory/domain_reachable` is the fully qualified metric name. In Observability Platform, the metric name is `gcp_cloudsql_database_cloudsql_googleapis_com_database_active_directory_domain_reachable`. ### Metric labels You can request custom labels for your Google Cloud metrics as `defaultLabels`. Contact [Chronosphere Support](/support) to add them. When importing metrics, some `defaultLabels` may conflict with prefixes which already exist in Observability Platform (for example, `job`). When this occurs, Observability Platform adds the prefix `exported_` to the source labels to prevent conflicts. ### Finding Google Cloud metrics in Metrics Explorer Use [Metrics Explorer](/investigate/querying/metrics/explorer) to find and review the status of your ingested metrics. * All Google Cloud metrics start with the prefix `gcp_`. Searching for this prefix displays all Google Cloud metrics in the platform. * Substring search is possible. For example, if the original Google Cloud metric name contains a substring like `domain_reachable`, searching for the substring returns the Google Cloud metric, along with other metrics containing the substring. ### Metric mappings Google Cloud metrics map closely to Observability Platform metric types. Metric mappings are defined on the [metric types](/control/shaping/shape-metrics/types#google-cloud) page. ### Metric prefix search Google Cloud integration scrapes metrics by metric prefix search using the [Monitoring Query Language (MQL)](https://cloud.google.com/monitoring/custom-metrics/reading-metrics). Metrics prefix searches use an implicit wildcard (`*`), and don't require wildcards in the configuration to search for multiple metrics. This following table provides configuration examples for Google Cloud integration: | Metric Prefix | Description | Example Metrics | | ------------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cloudsql.googleapis.com/` | Ingest all CloudSQL metrics | `cloudsql.googleapis.com/database/active_directory/domain_reachable` `cloudsql.googleapis.com/database/active_directory/instance_available` `cloudsql.googleapis.com/database/data_cache/bytes_used` `cloudsql.googleapis.com/database/uptime` | | `cloudsql.googleapis.com/database/` | Ingest all CloudSQL metrics starting with `database/` | `cloudsql.googleapis.com/database/active_directory/domain_reachable` `cloudsql.googleapis.com/database/active_directory/instance_available` `cloudsql.googleapis.com/database/data_cache/bytes_used` `cloudsql.googleapis.com/database/uptime` | | `bigquery.googleapis.com/s` | Ingest all BigQuery metrics starting with `s` | `bigquery.googleapis.com/slots/allocated` `bigquery.googleapis.com/slots/allocated_for_reservation` `bigquery.googleapis.com/storage/insertall_inserted_rows` `bigquery.googleapis.com/storage/uploaded_row_count` | | `cloudsql.googleapis.com/database/up` | Ingest all CloudSQL metrics starting with `database/up` | `cloudsql.googleapis.com/database/up` `cloudsql.googleapis.com/database/uptime` | | `b` | Ingest all metrics starting with `b` | `backupdr.googleapis.com/protected_data/volume` `bigquery.googleapis.com/slots/allocated` `bigtable.googleapis.com/backup/bytes_used` | # Ingest metrics and traces using the OpenTelemetry Collector Source: https://docs.chronosphere.io/ingest/metrics-traces/otel Use the OpenTelemetry Collector to collect metrics and traces and transfer them to Chronosphere for analysis. The [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) is vendor agnostic, open source, and supports popular backends and the OpenTelemetry Protocol. You can use the OpenTelemetry Collector to ingest metric data, and configure dynamic, remotely configurable [head sampling](/control/shaping/sample-traces/head-sampling#configure-the-opentelemetry-collector) for your trace data. ## Get started To get started with the OpenTelemetry Collector: 1. [Instrument your app](https://opentelemetry.io/docs/instrumentation/) with an [OpenTelemetry SDK](https://opentelemetry.io/docs/instrumentation/). 2. Create an API token to authenticate with Chronosphere Observability Platform. You must [create a service account](/administer/accounts-teams/service-accounts#create-a-restricted-service-account). Chronosphere recommends creating a restricted service account with a write-only scope. Use the generated API token in the OpenTelemetry Collector file `config.yml` to authenticate with Observability Platform. Store your API token in a secure location. If you lose your token, you must create a new service account. 3. Configure your OpenTelemetry Collector to [ingest metrics or traces](/ingest/metrics-traces/otel/otel-ingest). ## Conversion from OpenTelemetry to Prometheus-compatible metrics Observability Platform follows the [OpenTelemetry to Prometheus Specification](https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#otlp-metric-points-to-prometheus) to convert OpenTelemetry metrics to Prometheus-compatible metrics and adds delta temporality aggregation support to provide a more seamless experience for delta metrics. Observability Platform implements the following data conversions as defined in the [OpenTelemetry to Prometheus Specification](https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#otlp-metric-points-to-prometheus): * Sanitize metric and label names to conform to [Prometheus naming conventions](/ingest/metrics-traces/collector/mappings/prometheus/prometheus-recommendations). For example, an OpenTelemetry metric named `http.duration` becomes `http_duration`. * Collapse multiple consecutive underscore (`_`) characters to a single underscore character. * Metric names for OpenTelemetry explicit bucket boundary histograms follow the OpenMetrics specification to correctly name the time series for each histogram bucket. For example, a histogram has one `_bucket` series for each bucket, and a series for the `_sum` and `_count`. * Observability Platform supports staleness markers and writes them whenever any OpenTelemetry data point presents a `NoRecordedValue` flag for the associated scope. * Observability Platform **requires** `service.instance.id` for all metric time series to ensure [metric writer uniqueness](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#single-writer), and **rejects** metrics without a `service.instance.id` resource attribute. To configure a value for `service.instance.id`, follow the recommendations in [Map resource attributes to Prometheus job and instance](/ingest/metrics-traces/otel/otel-ingest#map-resource-attributes-to-prometheus-job-and-instance). Observability Platform deviates from the [OpenTelemetry to Prometheus Specification](https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#otlp-metric-points-to-prometheus) to reduce operational chores and improve metric usability. * Observability Platform preserves metric names and doesn't apply metric type or unit suffixes to metrics as defined in the OpenMetrics specification, with the exception of explicit bucket boundary histograms. * Observability Platform drops OpenTelemetry resource and data point attributes that have empty values from the time series. Observability Platform then accepts the resulting time series for processing. * Observability Platform merges both OpenTelemetry Protocol resource attributes and individual data point attributes into a single set of Prometheus labels for each time series. Merging resource attributes after ingestion removes the need to [manually configure resource attribute copying](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusexporter/README.md#setting-resource-attributes-as-metric-labels) in your OpenTelemetry Collector configuration. To configure resource attribute exclusions or turn off resource attribute merging, see [Configure OpenTelemetry ingestion](/ingest/metrics-traces/otel/otel-config). * Observability Platform doesn't create a [`target_info`](https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#resource-attributes-1) metric by default. The `target_info` metric is equivalent to the `up` metric, the presence of which indicates that a resource is available. You can [change your configuration](/ingest/metrics-traces/otel/otel-config) to enable `target_info` metric creation. # Ingest DogStatsD metrics with the OpenTelemetry Collector Source: https://docs.chronosphere.io/ingest/metrics-traces/otel/ingest-dogstatsd How to use the OpenTelemetry Collector to ingest DogStatsD metrics. The [OpenTelemetry Collector StatsD receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver) accepts, transforms, and aggregates DogStatsD metrics into OpenTelemetry protocol (OTLP) metrics for use in the Chronosphere Observability Platform. ## Install the OpenTelemetry Collector To install the OpenTelemetry Collector, deploy it as a Kubernetes DaemonSet on the same node as your DogStatsD clients. DogStatsD clients use the User Datagram Protocol (UDP) to send metrics. The OpenTelemetry Collector must be on the same node as the DogStatsD client to ensure reliable network communication between the DogStatsD client and the OpenTelemetry Collector. ## Configure the OpenTelemetry StatsD receiver You can configure the StatsD receiver by adding it to your [OpenTelemetry Configuration](https://opentelemetry.io/docs/collector/configuration/). The following configuration defines an OpenTelemetry receiver and a [StatsD receiver](https://github.com/statsd/statsd/blob/master/README.md) to accept DogStatsD metrics from Datadog clients. ```yaml theme={null} receivers: otlp: protocols: grpc: statsd/dogstatsd: # Default UDP port. endpoint: "localhost:8125" # Default interval. Use your licensed resolution: 60s, 30s or 15s. aggregation_interval: 60s # Metric type must be enabled. Default. enable_metric_type: true # DogStatsD clients support key-only tags, but attributes without a value are ignored in the Prometheus data model. enable_simple_tags: false # DogStatsD clients send delta counters (non-monotonic counters). is_monotonic_counter: true enable_ip_only_aggregation: true # Aggregates DogStatsD timers as exponential histograms. timer_histogram_mapping: - statsd_type: "timing" observer_type: "histogram" histogram: max_size: 160 - statsd_type: "histogram" observer_type: "histogram" histogram: max_size: 160 - statsd_type: "distribution" observer_type: "histogram" histogram: max_size: 160 ``` ## Configure an OpenTelemetry protocol exporter to send DogStatsD metrics DogStatsD metrics sent to Datadog are normalized to follow [Datadog's best practices](https://docs.datadoghq.com/developers/guide/what-best-practices-are-recommended-for-naming-metrics-and-tags/). For example, Datadog lowercases tag names and values and modifies tag values to begin with an alphabetical character. To reduce migration efforts and minimize changes to queries, Observability Platform can apply the same sanitization to ensure metric names, tags, and values are the same in both Observability Platform and Datadog by passing an additional header. The following configuration defines these OpenTelemetry protocol (OTLP) exporters: * `otlp/chronosphere` defines a default configuration to send OTLP metrics without applying Datadog normalization. Use this exporter to send all metrics from any source *except* Datadog clients. * `otlp/chronosphere/dogstatsd` defines a configuration to send OTLP metrics and apply Datadog normalization. Use this exporter only to send metrics that were sent by Datadog clients. ```yaml theme={null} exporters: # Exporter configured without normalization otlp/chronosphere: endpoint: TENANT:443 timeout: 30s retry_on_failure: enabled: true compression: zstd headers: API-Token: $API_TOKEN # Exporter configured to apply Datadog metric normalization otlp/chronosphere/dogstatsd: endpoint: TENANT:443 timeout: 30s retry_on_failure: enabled: true compression: zstd headers: API-Token: ${env:API_TOKEN} # Validation Response verbosity: SHORT, SUMMARY, DETAILED Chronosphere-Metrics-Validation-Response: SHORT # Enable Datadog metric name and tag normalization Chronosphere-Metrics-Translation: datadog Chronosphere-Metrics-Translation-Datadog-NormalizeLabelValues: "true" ``` To configure these exporters for your instance: 1. 2. specified as an HTTP header. Chronosphere recommends calling this value as an [environment variable](/ingest/metrics-traces/otel/otel-ingest#prerequisites). 3. Set `retry_on_failure` to `true` to retry attempts after any recoverable errors. 4. Set `timeout` to `30s` to prevent larger requests from timing out, since the upstream system might require more time for internal batching. ## Define OpenTelemetry Collector pipelines To receive metrics, connect the receivers and exporters with two separate pipelines: * **`metrics` pipeline:** The default metrics pipeline receives all OTLP metrics and sends them to the Chronosphere OTLP endpoint. * **`metrics/dogstatsd` pipeline:** The DogStatsD metrics pipeline receives DogStatsD metrics and exports them to a Chronosphere endpoint with a header to enable Datadog metric name and tag normalization. The following service configuration enables the StatsD receiver for the appropriate pipeline: ```yaml theme={null} service: pipelines: # Defines the default metrics pipeline. metrics: receivers: [otlp] processors: [batch, resourcedetection, resource/service-instance] exporters: [otlp/chronosphere] # Defines a metrics pipeline with Datadog metric normalization. metrics/dogstatsd: receivers: [otlp/chronosphere/dogstatsd] processors: [batch, resourcedetection, resource/service-instance] exporters: [otlp/chronosphere/dogstatsd] ``` The `processors` value adds processors to define batching and ensure time series uniqueness. * `batch`: Sets batch processing. For more information, see the "Configure batch processing" step in [Configure the OpenTelemetry Collector](/ingest/metrics-traces/otel/otel-ingest#configure-the-opentelemetry-collector). * `resourcedetection`: Collects information about the environment in which the OpenTelemetry Collector is running. * `resource/service-instance`: Observability Platform requires a `service.instance.id` attribute and value for all time series to ensure time series uniqueness. The ingestion API rejects metrics that lack `service.instance.id`. For more information, see the [Map resource attributes](/ingest/metrics-traces/otel/otel-ingest#map-resource-attributes-to-prometheus-job-and-instance). # Configure ingestion and processing of OpenTelemetry Metrics Source: https://docs.chronosphere.io/ingest/metrics-traces/otel/otel-config Learn how to configure Chronosphere Observability Platform's ingestion and processing of OpenTelemetry Metrics data For prerequisites to configuring the OpenTelemetry Collector for metrics data ingestion, see [Configure your OpenTelemetry Collector](/ingest/metrics-traces/otel/otel-ingest). ## OpenTelemetry ingestion configuration options The OpenTelemetry Protocol (OTLP) [metric data model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/data-model.md#opentelemetry-protocol-data-model) defines separate **Resource** and **Data Point** attribute contexts. * The [Resource context](https://opentelemetry.io/docs/concepts/resources/) contains attributes, such as `host.name` or `k8s.cluster.name`, that are common to all time series produced by the metric writer. * The Data Point context contains attributes unique to the time series, such as `http.route`. Chronosphere Observability Platform provides a configuration interface to control OpenTelemetry Metric Resource attribute processing. The default OpenTelemetry Metrics processing configuration in Observability Platform merges all Resource and Data Point attributes to the time series as labels. If there are conflicting attribute keys, Observability Platform uses the Data Point value. Observability Platform excludes some Resource attributes that are likely to exceed [attribute value size limits](/investigate/querying/metrics/troubleshooting#managing-invalid-metrics), which results in rejected time series. For details, see [Observability Platform-defined resource attribute exclusions](#observability-platform-defined-resource-attribute-exclusions). You can configure how Observability Platform processes OpenTelemetry Metrics to: * Prevent Observability Platform from merging resource attributes. * Enable the creation and writing of a `target_info` metric using all resource attributes. The `target_info` metric is functionally equivalent to the `up` metric, the presence of which indicates that a resource is available. * Change how Observability Platform handles conflicts when merging Resource and Data Point attributes. * Define an exclusion list of resource attribute keys to exclude from merging. ## Resource attribute configuration options The OpenTelemetry Metrics Ingestion configuration has a top-level `resource_attributes` field with additional sub-fields that define how Observability Platform processes resource attributes. | Field | Type | Description | Default value | | ---------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | `resource_attributes` | N/A | Top-level configuration field | N/A | | `generate_target_info` | Boolean | Determines whether to create a `target_info` metric. | `False` | | `flatten_mode` | Enumeration | Determines how Observability Platform merges resource attributes into the time series. | `MERGE` | | `filter_mode` | Enumeration | Determines which resource attributes Observability Platform processes when generating `target_info` and merging resource attributes into time series as labels. | `APPEND_DEFAULT_EXCLUDE_KEYS` | | `exclude_keys` | String array | Key names to exclude from both `target_info` generation and resource attribute merging. | [Default list](#observability-platform-defined-resource-attribute-exclusions) | ### `flatten_mode` The `flatten_mode` field determines how Observability Platform merges resource attributes into the time series. Its valid values are: * `MERGE` (default): Observability Platform merges Resource attributes into every data point associated with that resource, so that both sets end up as Prometheus labels on each individual metric time series. **If attribute keys conflict, Observability Platform uses the value from the data point.** * `IGNORE`: Observability Platform doesn't copy resource attributes to the resulting time series. * `OVERWRITE`: Observability Platform copies resource attributes onto every data point. **If attribute keys conflict, Observability Platform uses the value from the resource.** ### `filter_mode` The `filter_mode` field determines which resource attributes Observability Platform processes when generating `target_info` metrics and merging resource attributes into time series as labels. Its valid values are: * `APPEND_DEFAULT_EXCLUDE_KEYS` (default): Observability Platform includes all attributes unless they're defined in [your `exclude_keys` list](#exclude_keys) or the default [Observability Platform-defined resource attribute exclusions](#observability-platform-defined-resource-attribute-exclusions) list. With this setting, Observability Platform appends any `exclude_keys` you define to the default Observability Platform-defined list. * `CUSTOM_EXCLUDE_KEYS`: Observability Platform includes all attributes unless they're specifically defined in your `exclude_keys` list. With this setting, Observability Platform applies only your defined `exclude_keys` list. ### `exclude_keys` Use the `exclude_keys` field to list key names that you want Observability Platform to exclude from both `target_info` generation and resource attribute merging. The field's format is an array of strings, such as: ```text theme={null} [ "example.exclude", "another.example.exclude" ] ``` The resource attribute exclusions that Observability Platform applies by default aren't returned in this response. See [Observability Platform-defined resource attribute exclusions](#observability-platform-defined-resource-attribute-exclusions) for the latest list of default key exclusions. ## Observability Platform-defined resource attribute exclusions Some OpenTelemetry Resource attributes are likely to exceed Observability Platform [attribute value size limits](/administer/limits-licensing/limits/metric-limits), resulting in [rejected time series](/investigate/querying/metrics/troubleshooting#managing-invalid-metrics). Observability Platform excludes the following Resource attributes by default: | Rejected key | Reason | | ---------------------- | ---------------------------------------------------- | | `process.command_args` | Commonly exceeds the maximum attribute value length. | | `process.command_line` | Commonly exceeds the maximum attribute value length. | In cases where the OpenTelemetry Collector truncates these attributes to conform with Observability Platform ingestion requirements, you can change the merge behavior to include all resource attributes by setting `filter_mode = CUSTOM_EXCLUDE_KEYS`. ## Manage OpenTelemetry metrics ingestion configurations The Chronosphere API Config Endpoint provides methods to create, read, update, and delete the OpenTelemetry Metrics Configuration. ### View the configuration You can view OTLP ingestion configurations using one of the following methods. Use the `read` command of [Chronoctl](/tooling/chronoctl): ```shell theme={null} chronoctl otel-metrics-ingestion read ``` The command returns a YAML definition of the `OtelMetricsIngestion` resource. If the command returns an HTTP `404 Not Found` status code, your tenant uses the default configuration. To complete this action with the Chronosphere API, use the [`ReadOtelMetricsIngestion`](/tooling/api-info/definition/operations/ReadOtelMetricsIngestion) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Create or update the configuration You can change the default OpenTelemetry Metrics ingestion configuration using either [Chronoctl](/tooling/chronoctl) or the [Terraform provider](/tooling/infrastructure/terraform). Use the `scaffold` command to generate an example `OtelMetricsIngestion` resource. ```shell theme={null} chronoctl otel-metrics-ingestion scaffold ``` The following example enables `target_info` metric generation and defines two additional resource attribute key names to exclude. ```yaml Chronoctl example icon="square-terminal" theme={null} api_version: v1/config kind: OtelMetricsIngestion spec: resource_attributes: generate_target_info: true flatten_mode: MERGE filter_mode: APPEND_DEFAULT_EXCLUDE_KEYS exclude_keys: - example.exclude - another.example.exclude ``` Add the contents to a YAML file and send it to Observability Platform with the `apply` command: ```shell theme={null} chronoctl apply -f ``` You can also use the `otel-metrics-ingestion create` subcommand to create the resource, or the `otel-metrics-ingestion update` subcommand to update an existing resource: ```shell theme={null} chronoctl otel-metrics-ingestion create -f chronoctl otel-metrics-ingestion update -f ``` Replace `` with the filename of the YAML file containing the resource definition. Use the `chronosphere_otel_metrics_ingestion` resource, followed by a name in the resource declaration. The name you provide isn't important because Observability Platform allows only one OpenTelemetry Metrics ingestion configuration. The following example enables `target_info` metric generation and defines two additional resource attribute key names to exclude: ```terraform Terraform example icon="square-terminal" theme={null} resource "chronosphere_otel_metrics_ingestion" "my-otel-metrics-ingestion" { resource_attributes { flatten_mode = "MERGE" filter_mode = "APPEND_DEFAULT_EXCLUDE_KEYS" exclude_keys = ["example.exclude", "another.example.exclude"] generate_target_info = true } } ``` To create an OpenTelemetry ingestion configuration with the Chronosphere API, use the [`CreateOtelMetricsIngestion`](/tooling/api-info/definition/operations/CreateOtelMetricsIngestion) endpoint. To update an OpenTelemetry ingestion configuration with the Chronosphere API, use the [`UpdateOtelMetricsIngestion`](/tooling/api-info/definition/operations/UpdateOtelMetricsIngestion) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. ### Delete the configuration Use one of the following methods to delete an OpenTelemetry configuration. You can use [Chronoctl](/tooling/chronoctl) to delete your OpenTelemetry Metrics ingestion configuration and restore the default configuration. Use the `delete` command to delete the configuration. ```shell theme={null} chronoctl otel-metrics-ingestion delete ``` If the command fails because of a missing slug, use the `read` command to view the resource's slug. ```shell theme={null} chronoctl otel-metrics-ingestion read ``` To complete this action with the Chronosphere API, use the [`DeleteOtelMetricsIngestion`](/tooling/api-info/definition/operations/DeleteOtelMetricsIngestion) endpoint. Because the Chronosphere API requires authentication, include an API token with your `curl` request, as shown in the following example. For more details, see [Create an API token](/tooling/api-info#create-an-api-token). ```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null} export CHRONOSPHERE_API_TOKEN="TOKEN" export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io" curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \ -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH" ``` Replace the following: * *`TOKEN`*: Your API token. * *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance. * *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`. * *`ENDPOINT_PATH`*: The specific endpoint you want to access. # Configure your OpenTelemetry Collector Source: https://docs.chronosphere.io/ingest/metrics-traces/otel/otel-ingest Learn about configuring the OpenTelemetry Collector for metric and tracing data. You can configure the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to send metric and tracing data to Chronosphere Observability Platform. You must send all traffic over HTTPS. To learn more about OpenTelemetry, see [What is OpenTelementry?](https://chronosphere.io/learn/what-is-opentelemetry/) on the Chronosphere Blog. Observability Platform accepts OpenTelemetry Protocol (OTLP) metrics from OpenTelemetry Collectors by using the gRPC OTLP Exporter. Chronosphere recommends not sending OTLP metrics directly from the OpenTelemetry Client SDK. This procedure adds each configuration section separately. For a complete file, which you can modify as needed, see the [OpenTelemetry configuration example](#opentelemetry-configuration-example). ## Prerequisites To use the OpenTelemetry Collector, you need to [create a restricted service account](/administer/accounts-teams/service-accounts#create-a-restricted-service-account), which generates an API token you use to authenticate with Observability Platform. The access you provide for the service account depends on how you use the OpenTelemetry Collector. | Use case | Read access | Write access | | ------------------------------------------------------------------------- | :--------------------------: | :--------------------------: | | Collect metrics | | | | Remote [head sampling](/control/shaping/sample-traces/head-sampling) only | | | | Collect traces plus remote head sampling | | | Chronosphere recommends storing the API token generated from your service account in a separate file, or securely as a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/), and reference it using an environment variable, such as `$API_TOKEN`. Binding a Secret to an environment variable on the Pod is a well-supported, secure pattern in Kubernetes. ## Configure the OpenTelemetry Collector To configure the OpenTelemetry Collector to ingest metric and trace data: 1. Pull the [OpenTelemetry Docker image](https://opentelemetry.io/docs/collector/getting-started/#docker) to run the OpenTelemetry Collector in a Docker container: ```shell theme={null} docker pull otel/opentelemetry-collector-contrib:VERSION docker run otel/opentelemetry-collector-contrib:VERSION ``` Replace *`VERSION`* with the version of the OpenTelemetry Collector you want to run, which must be version 0.83 or later. 2. In the OpenTelemetry Collector [`config.yaml` file](https://opentelemetry.io/docs/collector/configuration/), apply the following settings to modify the `exporter` YAML collection. Specify an `endpoint` that points to your Observability Platform tenant, and include the Chronosphere API key you created as an HTTP header. You can modify the OpenTelemetry Collector [configuration](https://opentelemetry.io/docs/collector/configuration/) if you want to change the defaults. For example, you can reference a `config.yaml` in a different location, such as a Kubernetes ConfigMap. ```yaml theme={null} exporters: otlp/chronosphere: endpoint: TENANT:443 timeout: 30s retry_on_failure: enabled: true sending_queue: num_consumers: 50 # Valid values are snappy, gzip, zstd and none. compression: zstd headers: API-Token: ${env:API_TOKEN} Chronosphere-Metrics-Validation-Response: SHORT ``` * `endpoint`: * `timeout`: Set to `30s` to prevent larger requests from timing out, because the upstream system might require more time for internal batching. * `retry_on_failure`: Set `enabled` to `true` to enable retries for all retryable errors. * `sending_queue`: Set `num_consumers` to the number of consumers that dequeue batches from the sending queue. Increase the value of `num_consumers` if the exporter queue size demonstrates variability over time. A healthy sending queue metric depicts a small number of batches that shouldn't experience spikes over time. Chronosphere recommends setting this to 50 for [gateway deployments](https://opentelemetry.io/docs/collector/deployment/gateway/) of OTel Collectors. * `compression`: The compression method to apply. The Chronosphere OpenTelemetry endpoint supports `snappy`, `gzip`, and `zstd` compression, or `none`. * * *`Chronosphere-Metrics-Validation-Response`*: The `ErrorMessage` verbosity for rejected metrics. Default: `SHORT`. Valid values are: * `SHORT`, which reports the number of rejected metrics. * `SUMMARY`, which also includes counts of rejection reasons. * `DETAILED`, which also includes a sample of rejected metrics. 3. Configure [batch processing](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md). Sending telemetry in batches improves data compression and reduces the number of outgoing connections required to transmit the data. For example, the following configuration enforces a maximum batch size limit of `2000` spans without introducing any artificial delays: ```yaml theme={null} processors: batch: timeout: 1s send_batch_size: 1000 send_batch_max_size: 2000 ``` The `timeout`, `batch size`, and `batch max size` are default recommendations. Monitor the exporter send and enqueueing failure metrics to tune these parameters based on your workload. 4. Add the OTLP exporter and the batch processor to the metrics and traces exporters definition. ```yaml theme={null} service: pipelines: metrics: receivers: [otlp] processors: [batch] exporters: [otlp/chronosphere] traces: receivers: [otlp] processors: [batch] exporters: [otlp/chronosphere] ``` 5. Instruct the OpenTelemetry Collector to load the API token from an environment variable. Never share or store your API token in plain text. Chronosphere recommends using tools like SOPs to securely store this information. ```yaml theme={null} # In the OpenTelemetry Collector Deployment: - name: ${env:API_TOKEN} valueFrom: secretKeyRef: name: chronosphere-api-token key: apiToken # The accompanying Secret: apiVersion: v1 kind: Secret metadata: name: chronosphere-api-token namespace: YOUR_NAMESPACE type: Opaque data: apiToken: ${env:API_TOKEN} ``` 6. Save your OpenTelemetry Collector `config.yaml` file. ## Map resource attributes to Prometheus job and instance Observability Platform uses the OpenTelemetry `service.namespace`, `service.name` and `service.instance.id` resource attributes as the values for the Prometheus [job and instance source labels](https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#resource-attributes-1). * `service.namespace/service.name -> job`: Recommended. Observability Platform sets the job label value as a concatenation of `service.namespace` and `service.name` attribute values. If only `service.name` is set, then `job = "service.name"`. If neither `service.namespace` or `service.name` are present as resource attributes, then Observability Platform doesn't create a job label as part of the conversion. * `service.instance.id -> instance`: Required. Observability Platform sets the instance label value to the `service.instance.id` attribute value. Observability Platform rejects the metric if the `service.instance.id` resource attribute is missing. To ensure all metrics have a valid `service.instance.id` attribute, copy an existing, unique resource attribute, such as `host.name` or `pod.id`, as the unique instance identifier value for `service.instance.id`. To define a default `service.instance.id` resource attribute: 1. Add a resource detection processor. Use the resource detection processor can to detect resource information from the environment. Append or override the resource value in telemetry data with this information. Configure the resource detection processor based on the environment where you've deployed the Collector. Review the [resource detection processor documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor) to select the processor for your environment. For example, if the Collector is deployed on a host, the system processor is the best option to gather information about the host. ```yaml theme={null} processors: resourcedetection: detectors: [env, system] timeout: 2s override: true ``` 2. Define a resource attribute processor. In the processors ConfigMap, define a rule to add the `service.instance.id`. The following example maps the `host.name` resource attribute to a new `service.instance.id` resource attribute. Depending on how resource detection is configured, your environment might have other resource attributes available, such as `k8s.node.uid`, which can serve as the instance identifier. ```yaml theme={null} resource/service-instance: attributes: - key: service.instance.id from_attribute: host.name action: insert ``` 3. Add the resource detection and resource attribute processors to your metrics processors pipeline. In the `service.pipelines.metrics.processors` ConfigMap, add the resource attribute processor defined in the previous step: ```yaml theme={null} processors: [transform, batch, resourcedetection, resource/service-instance] ``` ## Send operational metrics about the OpenTelemetry Collector The OpenTelemetry Collector exposes metrics about its operations using a Prometheus scrape endpoint. Observability Platform uses OpenTelemetry Collector metrics in the OpenTelemetry Ingestion & Health dashboard to describe Collector health. To send operational metrics about your OpenTelemetry Collectors to Observability Platform: 1. Define a Prometheus receiver to scrape the endpoint receiver: ```yaml theme={null} prometheus/otel-collector-self-scrape: config: scrape_configs: - job_name: 'otel-collector-self-scrape' scrape_interval: 30s static_configs: - targets: ['0.0.0.0:8888'] ``` 2. Add the Prometheus receiver to the metrics pipeline: ```yaml theme={null} metrics: receivers: [prometheus/otel-collector-self-scrape, otlp/chronosphere] ``` 3. Add the metrics service to the Service ConfigMap: ```yaml theme={null} service: telemetry: metrics: address: "0.0.0.0:8888" ``` ## Next steps Observability Platform should begin ingesting data. Verify the Collector is receiving [metrics](/ingest/metrics-traces/collector/monitor/verify-metrics) and [traces](/ingest/metrics-traces/collector/monitor/verify-traces). You can also configure [head sampling](/control/shaping/sample-traces/head-sampling) to determine whether to drop a span or trace as early as possible. If you encounter issues, see the [troubleshooting](/ingest/metrics-traces/troubleshooting) page. ## OpenTelemetry configuration example The following example is the entire configuration file, containing all of the previously provided required sections: ```yaml filename="config.yaml" copy theme={null} receivers: # OTLP receiver otlp: protocols: grpc: http: # Prometheus receiver configured to scrape the Collector's own metrics prometheus/otel-collector-self-scrape: config: scrape_configs: - job_name: 'otel-collector-self-scrape' scrape_interval: 30s static_configs: - targets: ['0.0.0.0:8888'] exporters: # OTLP exporter configured to send telemetry to your Observability Platform tenant # Replace the endpoint's `MY_COMPANY` and `API_TOKEN` with your values otlp/chronosphere: endpoint: "TENANT:443" timeout: 30s retry_on_failure: enabled: true sending_queue: num_consumers: 50 # Compression method. Valid values are snappy, gzip, zstd, and none. compression: zstd headers: API-Token: ${env:API_TOKEN} # Validation Response verbosity. Valid values are SHORT, SUMMARY, and DETAILED. Chronosphere-Metrics-Validation-Response: SHORT processors: # Detect environment information to include as telemetry attributes # Configure the detector from the list of available detectors: # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor resourcedetection: detectors: [env, system] timeout: 2s override: true # Use environment information to define the `service.instance.id` attribute # This example maps `host.name` detected by the `resourcedetection` processor # to a new `service.instance.id` resource attribute resource/service-instance: attributes: - key: service.instance.id from_attribute: host.name action: insert # Add the batch processor to efficiently send telemetry batch: timeout: 1s send_batch_size: 1000 send_batch_max_size: 2000 service: pipelines: traces: receivers: [otlp] processors: [batch, resourcedetection, resource/service-instance] exporters: [otlp/chronosphere] metrics: receivers: [otlp] processors: [batch, resourcedetection, resource/service-instance] exporters: [otlp/chronosphere] # Define a separate metrics pipeline to send the Collector's metrics to # Observability Platform metrics/internal: receivers: [prometheus/otel-collector-self-scrape] processors: [batch, resourcedetection] exporters: [otlp/chronosphere] ``` # Use OpenTelemetry protocol endpoints Source: https://docs.chronosphere.io/ingest/metrics-traces/otel/otlp-endpoints Learn how to configure and use Chronosphere Observability Platform's endpoints for OpenTelemetry protocol ingestion of metrics and traces. Chronosphere Observability Platform supports ingestion of metrics and traces using OpenTelemetry protocol (OTLP) gRPC and HTTP ingestion endpoints. For best results, ingest telemetry through a local OpenTelemetry Collector for processing and batching. Sending telemetry directly from an OpenTelemetry SDK can be less reliable. ## Requirements Your OTLP configuration must meet certain requirements. * **`API-Token` header:** All requests must include the `API-Token` HTTP header, and its value must be the API token of a valid [service account](/administer/accounts-teams/service-accounts) with write permissions. * **OTLP version support:** Only v1.0 and higher are supported. * **Supported compression methods:** Chronosphere supports `gzip`, `snappy`, and `zstd` compression, and recommends `zstd`. * **Encryption:** All requests must be encrypted using TLS. - **Payload requirements:** * Compressed payloads should be smaller than 16 MB. * Metric payloads must contain fewer than 10,000 items. * For details, see [Optimize the batch processor](#optimize-the-batch-processor). ### Recommendations Chronosphere recommends setting the timeout value for a metrics or trace request to 30 seconds, and to enable retries to prevent transient network errors from causing data loss. Chronosphere recommends using gRPC instead of HTTP when possible, because gRPC is a more efficient protocol for sending telemetry to the Observability Platform. ### Optimize the batch processor The [OpenTelemetry Batch Processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor) defines batch limits by the quantity of items, not the size of the batch in bytes. Configure the batch processor as follows to prevent batches from exceeding the maximum payload size: * **`send_batch_size`:** Set to `1000` to trigger sending a batch when it reaches 1,000 items. * **`send_batch_max_size`:** Set to `2000` to force splitting the batch if it exceeds more than 2,000 items. * **`timeout`:** Set to `1s` to send any items in the buffer after 1 second. ## OTLP endpoint URLs Observability Platform provides gRPC and HTTP endpoints for metrics and traces. | Telemetry type | gRPC Endpoint | HTTP endpoint URL | | -------------- | ------------- | ---------------------------------------------- | | Metrics | `TENANT:443` | `https://TENANT/data/opentelemetry/v1/metrics` | | Traces | `TENANT:443` | `https://TENANT/data/opentelemetry/v1/traces` | OpenTelemetry exporters complete the path based on the telemetry type. For the `otlphttp` exporter, set the endpoint base path as follows: ```yaml theme={null} exporters: otlphttp/chronosphere: endpoint: https://TENANT/data/opentelemetry/ # ... ``` For an example OTLP exporter configuration, see [Configure the OpenTelemetry Collector](/ingest/metrics-traces/otel/otel-ingest#configure-the-opentelemetry-collector). ## Metrics partial success responses If a metrics ingestion request succeeds but any series are rejected, the ingester returns an [OTLP partial success](https://opentelemetry.io/docs/specs/otlp/#partial-success) response that includes a count (`rejected_data_points`) and a description (`error_message`). The verbosity of the description is controlled by the `Chronosphere-Metrics-Validation-Response` header, which can be set to `SHORT`, `SUMMARY`, or `DETAILED` in the [OTel Collector configuration](/ingest/metrics-traces/otel/otel-ingest#configure-the-opentelemetry-collector). When a partial success response is returned, the client must not retry the request. All valid series from the request have been accepted, and the rejected series are dropped. For details about the response's design, see the [OTLP Partial Success Specification](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc-response). # Troubleshoot ingestion Source: https://docs.chronosphere.io/ingest/metrics-traces/troubleshooting Troubleshoot issues with Observability Platform receiving data from Chronosphere and OTel Collectors. If you determine some or all of your data isn't displaying in Chronosphere Observability Platform, use the following information to ensure your Collector is ingesting metrics or traces. ## Verify incoming data Metrics and traces have different verification processes. ### Verify traces The [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer) provides a real-time view of incoming traces grouped by tag and their relative frequency. Use the Live Telemetry Analyzer to display the stream of incoming spans from your trace data. 1. In the navigation menu, click ** Go to Admin** and then select ** Analyzers > Live Telemetry**. 2. Click the **Traces** tab. 3. Click ** Capture live data** to display the stream of incoming spans. The default grouping shows spans by service (`__service__`), but you can include additional tags to group by, such as `__trace_id__` and `__span_id__`. ### Verify metrics You can also use the [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer) to inspect the incoming stream of metrics ingested by Observability Platform. 1. In the navigation menu, click ** Go to Admin** and then select ** Analyzers > Live Telemetry**. 2. Click the **Metrics** tab. 3. Click **Capture live data** to display streaming metrics. 4. In the **Keys** list, select the `__name__` and `instance` label keys. 5. In the **Values** filter, enter a `key:value` pair for a metric the OpenTelemetry exporter is sending, or any specific value the Collector is sending. The **OpenTelemetry Ingestion & Health** [dashboard](/observe/dashboards) provides two charts which show the rate of metric data points per received and rejected by the Chronosphere OTLP API. * **Metric Data Points Received** * **Metric Data Points Rejected** ## No data displays in Observability Platform Common reasons and corresponding log messages for data not appearing in Observability Platform: ### Missing API token If your API key is missing, the following message displays in the OpenTelemetry Collector logs: ```text wrap highlight=3 theme={null} Exporting failed. Try enabling retry_on_failure config option to retry on retryable errors {"kind": "exporter", "data_type": "traces", "name": "otlp/chronosphere", "error": "Permanent error: rpc error: code = Unauthenticated desc = missing auth header: API-Token"} ``` Ensure you've added the `API-Token` header to the `otlp/chronosphere` exporter [configuration](/ingest/metrics-traces/otel/otel-ingest) and that you're correctly setting and passing API token value. ### Invalid API token If your API key is invalid, the OpenTelemetry Collector logs report the following error: ```text wrap highlight=3 theme={null} Exporting failed. Try enabling retry_on_failure config option to retry on retryable errors {"kind": "exporter", "data_type": "traces", "name": "otlp/chronosphere", "error": "Permanent error: rpc error: code = Unauthenticated desc = invalid auth token"} ``` The Exporter is sending an `API-Token`, but it's invalid. Ensure you've correctly copied the API Token value and that you're correctly setting and passing the value in for the API token. ### Incorrect permission If your [service account](/administer/accounts-teams/service-accounts) doesn't have write permission, the OpenTelemetry Collector logs report the following error: ```text wrap highlight=3 theme={null} Exporting failed. Try enabling retry_on_failure config option to retry on retryable errors {"kind": "exporter", "data_type": "metrics", "name": "otlp/chronosphere", "error": "Permanent error: rpc error: code = PermissionDenied desc = Permission denied"} ``` The exporter requests an API token with write access. The exporter doesn't require read access, so you can resolve this issue by creating a new service account with a `write-only` scope. ## Some metrics are missing in Observability Platform The following sections can help you when you're ingesting some metrics, but some of the expected metrics are missing. Validations are applied to metrics as they're ingested. | Reason code | Description | Remediation | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `label_name_invalid` | The label name isn't compatible with Prometheus. The Collector and Chronosphere OTLP endpoint normalize this for you. | See the Prometheus documentation on [metric names and labels](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). | | `label_value_invalid` | The label value is invalid. This is enforced only for metric names, which must follow a specific format for PromQL compatibility. The Collector and Chronosphere OTLP endpoint normalize this on your behalf. | See the Prometheus documentation on [metric names and labels](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). | | `label_name_too_long` | The maximum supported length for a label name is 512 bytes. | Update your instrumentation or consider adding [Prometheus relabel rules](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) or OTel Collector processor changes. | | `label_count_too_high` | The maximum supported number of labels for a single time series is 64. | Update your instrumentation or consider adding [Prometheus relabel rules](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) or OTel Collector processor changes. | | `total_bytes_too_high` | The maximum supported amount of bytes used across all labels for a single time series is 4,096. | Update your instrumentation or consider adding [Prometheus relabel rules](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) or OTel Collector processor changes. | | `otel_service_instance_id_required` | See [`OTEL_SERVICE_INSTANCE_ID_REQUIRED` validation error](#otel_service_instance_id_required-validation-error). | See [`OTEL_SERVICE_INSTANCE_ID_REQUIRED` validation error](#otel_service_instance_id_required-validation-error). | | `histogram_sample_invalid` | An exponential histogram sample contains internal inconsistencies, such as a mismatch between the overall count field and the sum of all bucket counts. | Check for bugs in instrumentation libraries that emit exponential histograms or OTel Collector receivers or processors that might be corrupting histograms. | ## Metrics were dropped in a partial success When [ingesting metrics using the OpenTelemetry protocol](/ingest/metrics-traces/otel/otel-ingest), Observability Platform returns a [partial success](https://opentelemetry.io/docs/specs/otlp/#partial-success) response when [metrics are dropped](#metrics-were-dropped-in-a-partial-success). The OpenTelemetry Collector logs warnings for partial successes, and you can configure it to provide additional details about what failed. In the [OpenTelemetry Collector OTLP Exporter configuration](/ingest/metrics-traces/otel/otel-ingest#configure-the-opentelemetry-collector), add the `Chronosphere-Metrics-Validation-Response` header with the value set to `SUMMARY` or `DETAILED` to include the reasons why the metrics were dropped, and to see which metrics were dropped. The following is an example of a `DETAILED` response logged by the OpenTelemetry Collector: ```text wrap theme={null} 2025-05-09T17:42:29.363Z warn otlpexporter@v0.123.0/otlp.go:120 Partial success response {"message": "{\"message\":\"1 of 3 time series failed ingest validation and were dropped. This error includes a sample of up to 25 dropped series.\",\"details\":[{\"reason\":\"OTEL_SERVICE_INSTANCE_ID_REQUIRED\",\"metric\":\"testservice_metrics_sum\",\"subreason\":\"\",\"labels\":{\"deployment_environment\":\"local\",\"foo\":\"baz\",\"job\":\"testservice\",\"service_name\":\"testservice\"}}]}", "dropped_data_points": 1} ``` ## OTEL\_SERVICE\_INSTANCE\_ID\_REQUIRED validation error When using the OpenTelemetry Collector, Observability Platform requires `service.name` and `service.instance.id` as labels for all metrics to construct the Prometheus resource attributes. Metrics without these labels will be rejected and the following error message displays in your logs: ```text wrap theme={null} 2025-05-16T22:08:55.619Z warn otlpexporter@v0.126.0/otlp.go:120 Partial success response {"message": "78 of 78 time series failed ingest validation and were dropped. Add a header to the OTLP exporter to get additional details: 'Chronosphere-Metrics-Validation-Response: SUMMARY' or 'Chronosphere-Metrics-Validation-Response: DETAILED'", "dropped_data_points": 78} ``` Add the `Chronosphere-Metrics-Validation-Response: SUMMARY` header to the [OTLP Exporter](/ingest/metrics-traces/otel/otel-ingest#configure-the-opentelemetry-collector) to include the validation error code in the response. To resolve this, add an attributes processor to your metrics pipeline to map an existing label that can be used as the unique instance label for your environment. See [Mapping resource attributes](/ingest/metrics-traces/otel/otel-ingest#map-resource-attributes-to-prometheus-job-and-instance) for additional information. To troubleshoot this issue, investigate metric data points and their attributes to determine what attributes the metric includes. Add the `debug` exporter with `detailed` verbosity in the OpenTelemetry Collector configuration exporters ConfigMap. Then, add the `debug` exporter to your metrics pipeline. For example: ```yaml highlight={3-4} theme={null} exporters: # adds the debug exporter to print data points to log output debug: verbosity: detailed # ... snippet service: # ... snippet pipelines: metrics: receivers: [otlp] processors: [transform, batch, resourcedetection, resourceattributes/instance] exporters: [debug, otlp/chronosphere] ``` The following is sample output for a metric datapoint logged using the `debug` exporter. When you review the resource attributes list, you can see the required `service.name` is available, but `service.instance.id` is missing. The `host.name` attribute is a unique attribute which can be used as the `service.instance.id`. See [Mapping resource attributes](/ingest/metrics-traces/otel/otel-ingest#map-resource-attributes-to-prometheus-job-and-instance) to learn how to add attribute mapping to copy the `host.name` attribute to `service.instance.id`. ```text theme={null} 2023-10-18T16:38:46.172Z info ResourceMetrics #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1 Resource attributes: -> service.namespace: Str(opentelemetry-demo) -> service.name: Str(currencyservice) -> telemetry.sdk.version: Str(1.10.0) -> telemetry.sdk.name: Str(opentelemetry) -> telemetry.sdk.language: Str(cpp) -> host.name: Str(3d181cdaa016) -> os.type: Str(linux) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope app_currency 1.3.0 Metric #0 Descriptor: -> Name: app_currency_counter -> Description: -> Unit: -> DataType: Sum -> IsMonotonic: true -> AggregationTemporality: Delta NumberDataPoints #0 Data point attributes: -> currency_code: Str(USD) StartTimestamp: 2023-10-18 16:38:45.103115256 +0000 UTC Timestamp: 2023-10-18 16:38:46.106640465 +0000 UTC Value: 2 ``` # OpenTelemetry support in Observability Platform Source: https://docs.chronosphere.io/ingest/otel-ingestion Learn how Chronosphere Observability Platform supports OpenTelemetry ingestion for telemetry data. Chronosphere Observability Platform supports OpenTelemetry ingestion for logs, metrics, and traces. ```mermaid actions={false} theme={null} flowchart TD accTitle: OpenTelemetry support in Observability Platform accDescr: Use the OpenTelemetry SDK, OpenTelemetry Collector, and OTLP ingestion endpoints to send logs, metrics, and traces to Chronosphere Observability Platform. subgraph env [Your environment] SDK[OpenTelemetry SDK] Coll[OpenTelemetry Collector] StatsD[OpenTelemetry Collector
with StatsD receiver] end subgraph op [Observability Platform] OTLP[OTLP ingestion endpoints] end SDK -- Logs, metrics, and traces --> Coll SDK -- DogStatsD metrics --> StatsD Coll --> OTLP StatsD --> OTLP ``` This ingestion process varies depending on your environment, configuration settings, and telemetry type, but always follows the same general path: 1. Your app is instrumented with the OpenTelemetry SDK to emit telemetry data. 2. The OpenTelemetry Collector aggregates and processes this telemetry data. 3. Observability Platform ingests this telemetry data through the OpenTelemetry protocol (OTLP) ingestion endpoints. ## OpenTelemetry support for metrics You can use the [OpenTelemetry Collector](/ingest/metrics-traces/otel) to send metrics to Observability Platform. Observability Platform ingests these metrics through [OpenTelemetry protocol gRPC and HTTP ingestion endpoints](/ingest/metrics-traces/otel/otlp-endpoints). To ingest DogStatsD metrics, add the [StatsD receiver](/ingest/metrics-traces/otel/ingest-dogstatsd) to the OpenTelemetry Collector. When Observability Platform ingests metrics from OpenTelemetry, it [converts those metrics to a Prometheus-compatible format](/ingest/metrics-traces/otel#conversion-from-opentelemetry-to-prometheus-compatible-metrics). The [Live Telemetry Analyzer](/investigate/analyze/telemetry-analyzer) displays [special request metadata](/investigate/analyze/telemetry-analyzer#special-request-metadata) relevant to OpenTelemetry metrics. ## OpenTelemetry support for traces You can use the [OpenTelemetry Collector](/ingest/metrics-traces/otel) to send traces to Observability Platform. Observability Platform ingests these traces through [OpenTelemetry protocol gRPC and HTTP ingestion endpoints](/ingest/metrics-traces/otel/otlp-endpoints). You can also use the OpenTelemetry Collector to [configure head sampling](/control/shaping/sample-traces/head-sampling#configure-head-sampling) for traces. ## OpenTelemetry support for logs You can use the [OpenTelemetry Collector](/ingest/metrics-traces/otel) to send logs to Observability Platform. Observability Platform ingests these logs through an [OpenTelemetry protocol HTTP ingestion endpoint](/ingest/logs/otel-logs). ## Monitor your OpenTelemetry Collector health and ingestion volume When you configure the OpenTelemetry Collector to ingest metrics or traces, you can [send operational metrics about the OpenTelemetry Collector](/ingest/metrics-traces/otel/otel-ingest#send-operational-metrics-about-the-opentelemetry-collector) to Observability Platform. Chronosphere provides the [OpenTelemetry Ingestion & Health](/observe/dashboards/managed-dashboards#opentelemetry-ingestion-health) dashboard, which displays information about the health of your OpenTelemetry Collector instances and the amount of data they're ingesting and processing. # Chronosphere Telemetry Pipeline Source: https://docs.chronosphere.io/ingest/pipeline Learn about Chronosphere Telemetry Pipeline and when to use it to ingest data into Chronosphere. Chronosphere Telemetry Pipeline lets you control your observability data's journey from collection to processing to routing. Integrate with your existing observability tools to send logs, metrics, and traces from any source to any destination. ## Telemetry Pipeline v3 ## Telemetry Pipeline v2 ## General # Administer Telemetry Pipeline Source: https://docs.chronosphere.io/ingest/pipeline/administer Learn how to manage tools and features of Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. Manage teams, project settings, and other Chronosphere Telemetry Pipeline administration tasks. ## Get started # Manage project settings in Telemetry Pipeline Source: https://docs.chronosphere.io/ingest/pipeline/administer/project-settings Learn how to manage project settings for Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. Use the information on this page to manage settings for projects in Chronosphere Telemetry Pipeline. ## Enable or disable GPT generation GPT generation is enabled by default for new projects. > Requires Pipeline CLI version 3.81.0 or later. Certain parts of the Telemetry Pipeline web interface, like the configuration settings for the [custom Lua](/ingest/pipeline/processing-rules/custom-lua) processing rule, offer optional GPT generation tools. You can enable or disable these GPT generation tools on a per-project basis. To enable or disable GPT generation for a project, follow these steps in [Pipeline CLI](/ingest/pipeline/pipeline-cli): 1. Make sure your active [API token](/ingest/pipeline/pipeline-cli/authenticate) in Pipeline CLI is associated with the project where you want to enable or disable GPT generation. 2. Run the following command to update the project settings: ```shell theme={null} calyptia update project --set-setting gpt=VALUE ``` Replace *`VALUE`* with `true` to enable GPT generation or `false` to disable GPT generation. # Team members in Telemetry Pipeline Source: https://docs.chronosphere.io/ingest/pipeline/administer/team Add and manage team members in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. To add new team members to a project in Chronosphere Telemetry Pipeline, you can [send email invitations](/ingest/pipeline/administer/team/invite-members) or configure [single sign-on](/ingest/pipeline/administer/team/sso). Teams and [roles](#roles) are managed at the project level. If you're part of multiple projects in Telemetry Pipeline, adding or configuring a team member in one project won't affect your other projects. ## Roles Telemetry Pipeline offers several role types, each with a different set of permissions for reading, updating, creating, and deleting Telemetry Pipeline resources. These resources include any pipelines, Core Instances, plugins, processing rules, fleets, agents, and members associated with a project. The following table shows the different permissions available to each role: | | Viewer | Team | Manager | Admin | | ------ | ---------------------------- | ---------------------------- | ---------------------------- | ---------------------------- | | Read | | | | | | Update | | | | | | Create | | | | | | Delete | | | | | # Invite team members to your Telemetry Pipeline project Source: https://docs.chronosphere.io/ingest/pipeline/administer/team/invite-members Send invitations for new team members to join your project in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. You can send email invitations to add new members to a project in Chronosphere Telemetry Pipeline. 1. Sign in to [Telemetry Pipeline](https://core.calyptia.com/). 2. Open the project that you want to add members to. 3. Go to ** Team**, then click **Invite Members**. 4. In the **Email List** field, enter the email addresses of the members you want to invite. To invite multiple members, use commas to separate each email address. 5. Click **Invite**. All new members are assigned the viewer [role](/ingest/pipeline/administer/team#roles). by default. After a member accepts your invitation, you can configure their settings to assign them a different role. # Set up single sign-on Source: https://docs.chronosphere.io/ingest/pipeline/administer/team/sso Set up SSO for Telemetry Pipeline through your organization's identity provider. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. If your organization manages user credentials through an identity provider, you can use either SAML or OIDC to set up single sign-on (SSO) access to the Chronosphere Telemetry Pipeline web interface. ## SAML authentication To set up SAML authentication through your identity provider, you must: * Use the value `CONNECTION_NAME` for your connection name. * Set the single sign-on URL to `https://sso.calyptia.com/login/callback?connection=CONNECTION_NAME`. * Set the audience URI or SP entity ID to `urn:auth0:dev-15smjh-e:CONNECTION_NAME`. * Include the attributes `name`, `email`, and `email_verified` in your SAML assertion: * Use the value `user.firstName` for the `name` attribute. * Use the value `user.email` for the `email` attribute. * Use the value `true` for the `email_verified` attribute. * Download an X.509 certificate from your identity provider. After completing these tasks, notify Chronosphere that you've enabled a SAML connection, and include a copy of your X.509 certificate with your message. ## OIDC authentication To set up OIDC authentication through your identity provider, you must: * Set the sign-in redirect URIs to `https://sso.calyptia.com/login/callback` and `https://dev-15smjh-e.us.auth0.com/login/callback`. After completing this task, notify Chronosphere that you've enabled an OIDC connection. # Navigate the Telemetry Pipeline web interface Source: https://docs.chronosphere.io/ingest/pipeline/navigate Learn how to navigate the Chronosphere Telemetry Pipeline web interface. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. After you've [installed](/ingest/pipeline/v2/install) Chronosphere Telemetry Pipeline, you can navigate the [Telemetry Pipeline web interface](https://core.calyptia.com/) to access a variety of tools and features. For information about interacting with Telemetry Pipeline through a command-line interface, see [Pipeline CLI](/ingest/pipeline/pipeline-cli). ## Home The Telemetry Pipeline home page includes quick links to common pages and several embedded video tutorials. ## Core Instances > This page is only visible for v2 projects. This page lists the [Core Instances](/ingest/pipeline/v2/concepts#core-operators-and-core-instances) that you've created and displays information about each, including: * **Core Instance**: The name of that Core Instance. * **Status**: The status of that Core Instance. * **Pipelines Count**: The number of pipelines associated with that Core Instance. * **Version**: The version of the [Core Operator](/ingest/pipeline/v2/component-versions) associated with that Core Instance. * **Last Modification**: The date and time of the most recent changes to that Core Instance. You can click  **Deploy a new Core Instance** to [create a new Core Instance](/ingest/pipeline/v2/install/create-instance). To view more information about a specific Core Instance, to modify that Core Instance, or to view or modify any of its associated pipelines, click the name of that Core Instance. This opens the Core Instance details page. ### Core Instance details This page displays information about a specific Core Instance, including its name, tags, version, information about its Kubernetes environment, and any associated pipelines. Click **Edit** to modify the Core Instance's name and tags, or click  **Advanced Settings** to manage [files](/ingest/pipeline/v2/configure/files) and [secrets](/ingest/pipeline/v2/configure/secrets). In the **Kubernetes Namespaces** section, you can enable [automated Kubernetes logging](/ingest/pipeline/v2/build/automated-logging). The **Data Pipelines** section lists the pipelines within that Core Instance and displays information about each, including: * **Name**: The name of that pipeline. * **Status**: The status of that pipeline. * **Workload**: The [workload type](/ingest/pipeline/v2/configure/kubernetes/workloads) of that pipeline. * **Version**: The [version](/ingest/pipeline/v2/component-versions) of the pipeline agent powering that pipeline. If a pipeline isn't using the latest agent version, an **Upgrade** is displayed, which you can click to navigate to the [pipeline details](#pipeline-details-v2-pipelines) page. * **Uptime**: How long that pipeline has been running. * **Last modified**: The date and time of the most recent changes to that pipeline. You can also click **Create Data Pipeline** to [create a new pipeline](/ingest/pipeline/v2/build/create-modify). To view more information about a specific pipeline or to modify its settings, click the name of that pipeline. This opens the pipeline details page. ### Pipeline details (v2 pipelines) This page displays information about a specific pipeline, including its active sources, destination, and endpoints. Click **Edit** to open the pipeline builder, where you can add and configure [source plugins](/ingest/pipeline/plugins/source-plugins), [destination plugins](/ingest/pipeline/plugins/destination-plugins), and [processing rules](/ingest/pipeline/processing-rules) for that pipeline. Click  **Advanced Settings** to manage that pipeline's advanced settings, including: * [Replicas](/ingest/pipeline/v2/configure/scaling) * [Deployment strategy](/ingest/pipeline/v2/configure/deployment-strategies) * [Version](/ingest/pipeline/v2/component-versions) and [agent image type](/ingest/pipeline/v2/configure/agent-image) * [Files](/ingest/pipeline/v2/configure/files) * [Secrets](/ingest/pipeline/v2/configure/secrets) If a pipeline isn't using the latest agent [version](/ingest/pipeline/v2/component-versions), an `Upgrade to available` notice is displayed, and you can click **Upgrade** to upgrade the pipeline to the latest version. #### Pipeline output (v2 pipelines) In the **Pipeline Output** section, click **Get Latest Logs** to view operational logs generated by that pipeline, including status changes and error messages. If your pipeline has the [Standard output](/ingest/pipeline/plugins/destination-plugins/standard-output) destination plugin, the telemetry data routed through your pipeline will also print here. When you use **Pipeline Output** while the Standard output plugin is active, your telemetry data is temporarily cached in the Telemetry Pipeline backend. ## Pipelines > This page is visible only for v3 projects. This page lists the pipelines that you've created and displays information about each, including: * **Pipeline Name**: The name of that pipeline. * **Status**: The status of that pipeline. * **Version**: The version of the [pipeline collector](/ingest/pipeline/v3#deploy-the-pipeline-collector) associated with that pipeline. * **Uptime**: How long that pipeline has been running. * **Modified**: The date and time of the most recent changes to that pipeline. You can click  **New Pipeline** to [build a pipeline configuration file](/ingest/pipeline/v3#build-a-pipeline-configuration-file). To view more information about a specific pipeline, to view previous versions of its configuration file, or to view its associated agents, click the name of that pipeline to open its pipeline details page. ### Pipeline details (v3 pipelines) The **Pipeline details** page displays information about a specific pipeline, including its active sources, destination, and agents. Any changes you make from this page are for preview purposes only and won't be applied to your pipeline. To apply your changes, use the  **Copy Config** or  **Download Config** buttons to export your modified pipeline configuration file, and then use this modified configuration file to [update your pipeline](/ingest/pipeline/v3#update-a-pipeline). #### Pipeline output (v3 pipelines) In the **Pipeline Output** section, click **Get Latest Logs** to view operational logs generated by that pipeline, including status changes and error messages. If your pipeline has the [Standard output](/ingest/pipeline/plugins/destination-plugins/standard-output) destination plugin, the telemetry data routed through your pipeline will also print here. When you use **Pipeline Output** while the Standard output plugin is active, your telemetry data is temporarily cached in the Telemetry Pipeline backend. ## Monitoring > This page is visible only for v2 projects. This page displays information about the performance and throughput of your Core Instances and pipelines. ## Fleets > This page is only visible for v2 projects. This page lists the [fleets](/ingest/pipeline/v2/fleets) that you've deployed and displays information about each, including: * **Fleet Name**: The name of that fleet. * **Agent Type**: The [type of agent](/ingest/pipeline/v2/fleets/agent) that makes up that fleet. * **Managed Agents**: The number of agents within that fleet. * **Last Edit**: The date and time of the most recent changes to that fleet. You can click  **Create new fleet** to [create a new fleet](/ingest/pipeline/v2/fleets#create-a-fleet). To view more information about a specific fleet, modify that fleet, or view details about its associated agents, click the name of that fleet. This opens the fleet details page. ### Fleet details This page displays information about a specific fleet, including its name, configuration settings, attached files, and any associated agents. The **Agents List** section lists the agents within that fleet and displays information about each, including: * **Host Name**: The name of the machine where that agent is deployed. * **Status**: The status of that agent. * **Version**: The version of the Fluent Bit or Core Agent for that agent. * **Platform**: The operating system where the agent is deployed. * **Architecture**: The architecture of the environment where the agent is deployed. * **Last Edit**: The date and time of the most recent changes to that agent. You can click  **Attach New File** to add a file to that fleet, or  **Add Agent** to [add a new agent](/ingest/pipeline/v2/fleets/add-agent). ## Parser playground The parser playground lets you preview the effect of [parsers](/ingest/pipeline/v2/build/parsers) on your telemetry data. The **Examples** column includes several of the default parsers included with Telemetry Pipeline, and you can test your own custom parsers by pasting any string into the **Regular Expression** field. To preview the result of a specified parser, either enter raw log data into the **Enter Your Test String** field or use the pre-populated data, then click **Parse**. The **Output** pane displays the resulting structured data. After you've created and testing a custom parser in the parser playground, you can [add that parser to a `parsers.conf` file](/ingest/pipeline/v2/build/parsers#custom-parsers). ## Pipeline builder playground > This page is only visible for v2 projects. The pipeline builder playground lets you construct pipelines in a sandbox environment. You can use this tool to configure [source plugins](/ingest/pipeline/plugins/source-plugins), [destination plugins](/ingest/pipeline/plugins/destination-plugins), and [processing rules](/ingest/pipeline/processing-rules), and then preview the resulting pipeline. The **Advanced Settings** section also generates a [YAML configuration file](/ingest/pipeline/v2/configure/config-files) from your choices. This tool is functionally identical to the pipeline builder for real production environments, but any plugins or processing rules you configure here won't go into effect and won't affect your telemetry data. If you'd like to export your choices for later use, including in a production environment, click **Export** to save a copy of the pipeline's configuration file to your device. ## Processing rules playground The processing rules playground lets you create and test [processing rules](/ingest/pipeline/processing-rules) in a sandbox environment, including custom Lua scripts. This tool is functionally identical to the processing rules builder for real production environments, but the playground environment is isolated from the internet and has no access to internal Telemetry Pipeline resources. Any processing rules you create here won't affect your pipelines, Core Instances, or telemetry data. If you'd like to save your playground processing rules for later use, including in a production environment, click  **Export** to save a JSON template file to your device. You can also click  **Save** to save your template within the Telemetry Pipeline web interface, which makes that template available to any other users in your current project. ## Settings This page displays information about your current Telemetry Pipeline project. You can also [generate new API keys](/ingest/pipeline/pipeline-cli/authenticate) and view keys that you previously generated. ## Team This page displays information about the team members in your current project, including: * **User Name**: The name of that user. * **Email**: The email address of that user. * **Permissions**: The permissions assigned to that user. * **Added on**: The time elapsed since that user was added to the project. Click **Invite Members** to invite members to your project, or click the  **Edit** icon next to a user to adjust settings for that user. ### Permissions The following permission levels are available for Telemetry Pipeline users: * **Viewer**: Users who can view your organization's Telemetry Pipeline resources. * **Team**: Users who can view and update Telemetry Pipeline resources. * **Manager**: Users who can view, update, and create Telemetry Pipeline resources. * **Admin**: Users who can view, update, create, and delete Telemetry Pipeline resources. # Pipeline CLI Source: https://docs.chronosphere.io/ingest/pipeline/pipeline-cli Learn about the Chronosphere Telemetry Pipeline command-line tool. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. Pipeline CLI is a Chronosphere-supplied app that lets you manage Chronosphere Telemetry Pipeline from the command line. ## Get started To install Pipeline CLI, [Install Pipeline CLI](/ingest/pipeline/pipeline-cli/install). After you install Pipeline CLI, you'll need to [authenticate](/ingest/pipeline/pipeline-cli/authenticate) with Chronosphere. ## View command list To view a full list of available commands and flags, run: ```shell theme={null} calyptia help ``` You can also prepend `calyptia help` to a specific command to view more information about that command. Pipeline CLI supports different commands for Telemetry Pipeline v2 and Telemetry Pipeline v3. Depending on whether the [active project](/ingest/pipeline/pipeline-cli/authenticate#view-active-project) is a v2 project or a v3 project, running `calyptia help` displays commands relevant to that version. ## View current version To view the version of Pipeline CLI that you currently have installed, run the following command: ```shell theme={null} calyptia version ``` ## Command autocompletion > This feature is compatible only with v2 projects. You can add support for shell command autocompletion with Pipeline CLI for Bash, Zsh, Fish, and PowerShell by using the `calyptia completion` command. This section provides detailed steps for Bash and Zsh on macOS. For Linux-specific details and instructions for Fish and PowerShell, see the output of `calyptia completion --help`. 1. Update or install `bash-completion`: ```shell theme={null} brew install bash-completion ``` 2. In a shell initialization script, such as `~/.bashrc`, add the following: ```shell theme={null} source "$(brew --prefix bash-completion)/etc/bash_completion" eval "$(calyptia completion bash)" ``` 3. Restart the shell. 1. Go to your Zsh `site-functions` folder. * If you installed Zsh with Homebrew, the path is `/opt/homebrew/share/zsh/site-functions`. * If you use the default macOS Zsh, the path is `/usr/local/share/zsh/site-functions`. 2. Run the following command: ```shell theme={null} calyptia completion zsh > _calyptia ``` 3. Ensure your `~/.zshrc` file has [`compinit`](https://zsh.sourceforge.io/Doc/Release/Completion-System.html) added. If you use `oh-my-zsh`, it adds that for you. 4. Restart the shell. You can now press `Tab` on the command line to automatically complete Pipeline CLI commands. # Authenticate in Pipeline CLI Source: https://docs.chronosphere.io/ingest/pipeline/pipeline-cli/authenticate Learn how to authenticate with Chronosphere through Pipeline CLI. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. To use Pipeline CLI, you must [generate an API token](#generate-an-api-token) to authenticate with Telemetry Pipeline, and then [set that token](#set-or-switch-an-api-token) in Pipeline CLI. API tokens are tied to specific [projects](/ingest/pipeline/administer/project-settings) within Telemetry Pipeline, which means only one API token can be active in your Pipeline CLI session. To switch between projects, you must [switch tokens](#set-or-switch-an-api-token) accordingly. ## Generate an API token Follow these steps to generate an API token in Telemetry Pipeline: 1. Sign in to the [Telemetry Pipeline web interface](https://core.calyptia.com/). 2. Open the project where you want to create a fleet. 3. In the navigation menu, select ** Settings**. 4. In the **Generate API key** section, enter a name for your new API key, and then click **Generate**. 5. If you have multiple projects in Telemetry Pipeline, repeat these steps for each project. ## Set or switch an API token Only one API token can be active in your Pipeline CLI session. Each API token is tied to a specific [project](/ingest/pipeline/administer/project-settings), which means that setting an API token determines which project is active. If you have multiple Telemetry Pipeline projects, follow these steps to switch between them. 1. Copy the value of the API token you [generated](#generate-an-api-token) for that project. 2. Run the following command in Pipeline CLI: ```shell /TOKEN/ theme={null} calyptia config set_token TOKEN ``` Replace *`TOKEN`* with the value of the API token you copied in the previous step. ### View active project To view the name of the project associated with your active API token, run the following command: ```shell theme={null} calyptia get project ``` # Install Pipeline CLI Source: https://docs.chronosphere.io/ingest/pipeline/pipeline-cli/install Learn how to install Pipeline CLI for Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. Pipeline CLI is compatible with Linux, macOS, and Windows for both arm64 and x86\_64 architectures. Use one of the following methods to install Pipeline CLI: | Method | Linux | macOS | Windows | | --------------- | :--------------------------: | :--------------------------: | :--------------------------: | | Homebrew | | | | | curl | | | | | Direct download | | | | If you used the `install-core.sh` script to [install a Core Operator on Linux](/ingest/pipeline/v2/install/install-operator/linux), the script installed Pipeline CLI for you. You don't need to install it again. ## Download and install Pipeline CLI Select from the following methods to download and install the latest version of Pipeline CLI. To install Pipeline CLI using Homebrew, run the following commands: ```shell theme={null} brew tap calyptia/tap brew install calyptia ``` These commands fetch the latest version of Pipeline CLI. To install Pipeline CLI using curl, run the following command, which is the same for both Zsh and Bash: ```shell theme={null} curl -sSfl https://raw.githubusercontent.com/calyptia/cli/main/install.sh | bash ``` This command fetches the latest version of Pipeline CLI. To install Pipeline CLI by downloading its files directly: 1. Go to the [Pipeline CLI release page](https://github.com/chronosphereio/calyptia-cli/releases). 2. Locate the latest release artifact and download the `.tar.gz` archive that corresponds to your operating system and architecture. 3. Extract the Pipeline CLI binary from the archive that you downloaded. 4. Run the executable file to install Pipeline CLI. Alternatively, you can download the Pipeline CLI source code from its release page and compile it yourself. ## Update Pipeline CLI To update Pipeline CLI, use any of the previous methods to run the Pipeline CLI installation process again. If you use the direct download method, be sure to download the release artifact for the latest version of Pipeline CLI. # Telemetry Pipeline plugins Source: https://docs.chronosphere.io/ingest/pipeline/plugins Learn how to use plugins in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. ```mermaid actions={false} theme={null} flowchart LR accTitle: Pipeline flow accDescr: A simplified representation of a telemetry pipeline, which includes a source plugin, parser, processing rules, and destination plugin. subgraph Pipeline flow direction LR A(Source plugin) B(Parser) C(Processing rules) D(Destination plugin) A --> B --> C --> D style A stroke-width:4px style D stroke-width:4px end ``` Chronosphere Telemetry Pipeline provides *plugins* that let you route telemetry data from one or more sources to one or more destinations. Plugins are customizable and offer support for logs, metrics, and traces, although not all plugins support all telemetry types. [Source plugins](/ingest/pipeline/plugins/source-plugins) ingest data into your pipelines, and [destination plugins](/ingest/pipeline/plugins/destination-plugins) send that data to the storage or analysis platform of your choice. ## Add plugins to a pipeline The source and destination plugins for each pipeline are listed in the `inputs` and `outputs` section of that pipeline's [configuration file](/ingest/pipeline/v2/configure/config-files). To add plugins to a pipeline, you can either edit a pipeline through the Telemetry Pipeline web interface, or edit the pipeline's configuration file directly. Each plugin includes a number of customizable settings, which are represented as key-value pairs within a configuration file. Many of these settings have default values. For example, this snippet shows the default configuration settings for an [OpenTelemetry source plugin](/ingest/pipeline/plugins/source-plugins/opentelemetry): ```yaml theme={null} pipeline: inputs: - Name: opentelemetry port: "8088" tag_from_uri: "false" buffer_chunk_size: 512K buffer_max_size: 4M successful_response_code: "201" tls.verify: on tls.debug: "1" <...> ``` ### Descriptive names To help keep track of your plugins, Telemetry Pipeline lets you add custom metadata to each plugin, including descriptive names. These descriptive names don't overwrite the default names assigned to each plugin. Instead, they're displayed under the plugin's default name in the Telemetry Pipeline web interface. To add a descriptive name to a plugin through the Telemetry Pipeline web interface: 1. In the **Pipeline Builder**, click the existing plugin to edit it. 2. Expand the **Metadata** section, and enter a value in the **Name** field. 3. Click **Save**. To add a descriptive name to a plugin by editing a [pipeline configuration file](/ingest/pipeline/v2/configure/config-files): * Add the following lines of YAML to the entry for that plugin: ```yaml theme={null} core.metadata: name: NAME ``` Replace *`NAME`* with the descriptive name you want to use for the plugin. The `core.metadata` key must be nested at the same level as other top-level configuration keys, and must *not* replace the default `Name` key. For example, this snippet shows an OpenTelemetry source plugin correctly configured to use a descriptive name: ```yaml theme={null} pipeline: inputs: - Name: opentelemetry port: "8088" core.metadata: name: otel-instance-1 <...> ``` # Send telemetry data with destination plugins Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins Configure Chronosphere Telemetry Pipeline to send telemetry data to various destinations. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. ```mermaid actions={false} theme={null} flowchart LR accTitle: Pipeline flow accDescr: A simplified representation of a telemetry pipeline, which includes a source plugin, parser, processing rules, and destination plugin. subgraph Pipeline flow direction LR A(Source plugin) B(Parser) C(Processing rules) D(Destination plugin) A --> B --> C --> D style D stroke-width:4px end ``` Chronosphere Telemetry Pipeline uses *destination plugins* to send telemetry data from your pipeline to various destinations. You can add and configure destination plugins to connect your pipelines to a variety of storage and analysis tools. Different destination plugins support different data types, including logs, metrics, and traces. # Amazon CloudWatch Logs and Metrics destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/amazon-cloudwatch Configure the Amazon CloudWatch Logs and Metrics destination plugin in Chronosphere Telemetry Pipeline to send logs and metrics to Amazon CloudWatch. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Amazon CloudWatch Logs and Metrics [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `cloudwatch_logs`) lets you configure your telemetry pipeline to stream your log data and metrics to CloudWatch Logs and Metrics, respectively. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ------------------------------ | ----------------- | ---------------------------------------------------------------------------- | ----------- | | **Region** | `region` | Required. The AWS region to which to send your data. | `us-east-1` | | **CloudWatch Log Group Name** | `log_group_name` | Required. CloudWatch Log Group Name. | *none* | | **CloudWatch Log Stream Name** | `log_stream_name` | Required. CloudWatch Log Stream Name. Not compatible with Log Stream Prefix. | *none* | ### AWS Authentication | Name | Key | Description | Default | | ---------------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------- | | **IAM Role ARN** | `role_arn` | Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role to assume, for uses like cross-account access. | *none* | | **CloudWatch Logs and Metrics Endpoint** | `endpoint` | Custom endpoint for the Firehose API. | *none* | | **STS API Endpoint** | `sts_endpoint` | Custom endpoint for the STS API. | *none* | | **External ID for STS API** | `external_id` | Specifies an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID. | *none* | ### Advanced | Name | Key | Description | Default | | -------------------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **CloudWatch Log Stream Name Prefix** | `log_stream_prefix` | Prefix for CloudWatch Log Stream Name. Tag is appended to the prefix to form the stream name. | *none* | | **CloudWatch Log Group Template** | `log_group_template` | Template for CloudWatch Log Group name using record accessor syntax. If needed, the plugin falls back to the `log_group_name` value. | *none* | | **CloudWatch Log Stream Template** | `log_stream_template` | Template for CloudWatch Log Stream name using record accessor syntax. If needed, the plugin falls back to the `log_stream_name` or `log_stream_prefix` value. | *none* | | **Number of days to retain logs** | `log_retention_days` | Any newly created log group's retention policy is set to this many days. Accepted values: `0`, `1`, `3`, `5`, `7`, `14`, `30`, `60`, `90`, `120`, `150`, `180`, `365`, `400`, `545`, `731`, `1827`, `3653`. | `0` | | **Log Format** | `log_format` | Used to tell CloudWatch the format of the data. A value of `json/emf` enables CloudWatch to extract custom metrics embedded in a JSON payload. | *none* | | **Log Key** | `log_key` | By default, the entire log record is sent to CloudWatch. If you specify a key name with this option, only the value of that key is sent to CloudWatch. For example, if you're using the Fluentd Docker log driver, you can specify `log_key log`, and only the log message is sent to CloudWatch. | *none* | | **Enable Auto Retry Requests** | `auto_retry_requests` | Whether to immediately retry failed requests to AWS services one time. This option doesn't affect the normal Telemetry Pipeline retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient or random networking issues. Accepted values: `true`, `false`. | `false` | | **Enable Auto Create Log Group** | `auto_create_group` | Whether to automatically create the log group. Log streams are always automatically created. Accepted values: `true`, `false`. | `false` | | **Metric namespace for CloudWatch EMF logs** | `metric_namespace` | Metric namespace for CloudWatch EMF logs. | *none* | | **Metric Dimension List** | `metric_dimensions` | Metric dimensions is a list of lists. If you have only one list of dimensions, put the values as a comma-separated string. If you want to put list of lists, use the list as semicolon-separated strings. If your value is `d1,d2;d3`, the plugin considers it as `[d1, d2], [d3]`. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Amazon S3 destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/amazon-s3 Configure the Amazon S3 destination plugin in Chronosphere Telemetry Pipeline to send logs to Amazon S3. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Amazon S3 [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `s3`) lets you configure your telemetry pipeline to store and archive your data by sending it directly to your Amazon S3 bucket. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | --------------------------- | ----------------- | -------------------------------------------------------------------------- | ----------- | | **Region** | `region` | Required. The AWS region to which to send your data. | `us-east-1` | | **Bucket** | `bucket` | S3 bucket name. | *none* | | **Total File Size (Bytes)** | `total_file_size` | Specifies the size of files in S3. Maximum size is 50 GB, minimum is 1 MB. | `1000000` | | **Compression** | `compression` | Sets the compression type. Accepted values: `arrow`, `zip`. | *none* | ### AWS Authentication | Name | Key | Description | Default | | ------------------------------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------- | | **AWS Shared Credential File** | `aws_shared_credentials_file` | Specifies the [shared credential file](#shared-credential-file) to use when uploading if not using AWS ARN. | *none* | | **IAM Role ARN** | `role_arn` | Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role to assume, for uses like cross-account access. | *none* | | **S3 Object ACL Policy** | `canned_acl` | Predefined Canned ACL policy for S3 objects. | *none* | | **S3 API Endpoint** | `endpoint` | Custom endpoint for the AWS S3 API. | *none* | | **STS API Endpoint** | `sts_endpoint` | Custom endpoint for the STS API. | *none* | | **External ID for STS API** | `external_id` | Specifies an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID. | *none* | ### Advanced | Name | Key | Description | Default | | -------------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- | | **Use Put Object** | `use_put_object` | Whether to use the S3 PutObject API, instead of multipart upload API. Accepted values: `true`, `false`. | `false` | | **Send Content-MD5 header** | `send_content_md5` | Sends the Content-MD5 header with object uploads as is required when Object Lock is Enabled. Accepted values: `true`, `false`. | `false` | | **Preserve Data Ordering** | `preserve_data_ordering` | Normally, when an upload request fails, there is a high chance for the last received chunk to be swapped with a later chunk, resulting in data shuffling. This feature prevents this shuffling by using a queue logic for uploads. Accepted values: `true`, `false`. | `false` | | **Log Key** | `log_key` | By default, the entire log record is be sent to S3. If you specify a key name with this option, only the value of that key is sent to S3. | *none* | | **Storage Class** | `storage_class` | Specifies the storage class for S3 objects. If this option isn't specified, objects are stored with the default `STANDARD` storage class. | *none* | | **Store Dir** | `store_dir` | Directory to locally buffer data before sending. Plugin uses the S3 Multipart upload API to send data in chunks of 5 MB at a time. Only a small amount of data is locally buffered at any given point in time. | `/tmp/calyptia-core/s3` | | **S3 Key Format** | `s3_key_format` | Format string for keys in S3. This option supports `strftime` time formats and a syntax for selecting parts of the Fluent log tag using a syntax inspired by the `rewrite_tag` filter. Add `$TAG` in the format string to insert the full log tag. Add `$TAG\[0]` to insert the first part of the tag in the S3 key. The tag is split into parts using the characters specified with the `s3_key_format_tag_delimiters` option. Add `$INDEX` to enable sequential indexing for file names, which prevents random string being added to end of the key when the `$UUID` isn't provided. | `/calyptia-core-logs/$TAG/%Y/%m/%d/%H/%M/%S` | | **S3 Key Format Tag Delimiters** | `s3_key_format_tag_delimiters` | A series of characters used to split the tag into parts for use with the `s3_key_format` option. | `.` | | **Use Static File Path?** | `static_file_path` | Disables behavior where the UUID string is automatically appended to end of the S3 key name when `$UUID` isn't provided in `s3_key_format`. `$UUID`, time formats, `$TAG`, and other dynamic key formats all work as expected when this feature is set to `true`. Accepted values: `true`, `false`. | `false` | | **Enable Auto Retry Requests** | `auto_retry_requests` | Whether to immediately retry failed requests to AWS services one time. This option doesn't affect the normal Telemetry Pipeline retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient or random networking issues. Accepted values: `true`, `false`. | `true` | | **JSON Date Format** | `json_date_format` | Specifies the format of the date. Accepted values: `double`, `epoch`, `iso8601` (for example, `2018-05-30T09:39:52.000681Z`), `java_sql_timestamp` (for example, `2018-05-30 09:39:52.000681`, which can be used with AWS Athena). | *none* | | **JSON Date Key** | `json_date_key` | Specifies the name of the date field in output. | *none* | | **Upload Chunk Size (Bytes)** | `upload_chunk_size` | This plugin uses the S3 Multipart Upload API to stream data to S3, which helps speed up data uploads. This parameter configures the size of each `part` in the upload. The `total_file_size` option configures the size of the file in S3. This option determines the size of chunks uploaded until that size is reached. These chunks are temporarily stored in `chunk_buffer_path` until their size reaches `upload_chunk_size`, at which point the chunk is uploaded to S3. Maximum value is 50 MB, minimum is 5 MB. | `5242880` | | **Upload Timeout** | `upload_timeout` | Optionally specifies a timeout for uploads. Whenever this amount of time has elapsed, Telemetry Pipeline completes an upload and creates a new file in S3. For example, set this value to 60m to get a new file in S3 every hour. | `10m` | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ## Shared credential file Your shared credential file provides authentication credentials to the Amazon S3 destination plugin. This file must be an [AWS credentials file](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html#cli-configure-files-format) that includes an `aws_access_key_id` parameter and an `aws_secret_access_key` parameter. For example: ```shell filename="Credentials" theme={null} [default] aws_access_key_id=ASIAIOSFODNN7EXAMPLE aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY ``` To reference this file in your plugin configuration, use the following syntax: ```yaml theme={null} {{ files.NAME }} ``` Replace *`NAME`* with the name of your credentials file. ## Bucket policies To use the Amazon S3 destination plugin, you must grant the plugin write access to your S3 buckets. These [bucket policies](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html) are managed within Amazon S3, not within the plugin's own configuration. For example, the following bucket policy lets the Amazon S3 destination plugin send data to a bucket named `my-bucket`: ```json filename="Bucket policy" theme={null} { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": "s3:PutObject", "Resource": [ "arn:aws:s3:::my-bucket", "arn:aws:s3:::my-bucket/*" ] }] } ``` # Amazon Kinesis Firehose destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/aws-firehose Configure the Amazon Kinesis Firehose destination plugin in Chronosphere Telemetry Pipeline to send logs to Amazon Kinesis Firehose. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Amazon Kinesis Firehose [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `kinesis_firehose`) lets you configure your telemetry pipeline to collect various data types such as logs, metrics, traces, and events and stream them directly to your Kinesis Firehose delivery stream. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | --------------------------------- | ----------------- | ---------------------------------------------------- | ----------- | | **Region** | `region` | Required. The AWS region to which to send your data. | `us-east-1` | | **Firehose delivery stream name** | `delivery_stream` | Kinesis Firehose name. | *none* | ### AWS Authentication | Name | Key | Description | Default | | --------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------- | | **IAM Role ARN** | `role_arn` | Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role to assume, for uses like cross-account access. | *none* | | **Firehose API Endpoint** | `endpoint` | Custom endpoint for the Firehose API. | *none* | | **STS API Endpoint** | `sts_endpoint` | Custom endpoint for the STS API. | *none* | | **External ID for STS API** | `external_id` | Specifies an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID. | *none* | ### Advanced | Name | Key | Description | Default | | ------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | | **Compression** | `compression` | Compression type for Firehose records. Each log record is individually compressed and sent to Firehose. Accepted values: `gzip`, `arrow` (with `arrow` available only if Apache Arrow is enabled at compile time). | *none* | | **Time Key** | `time_key` | Adds the timestamp to the record under this key. By default, the timestamp from Telemetry Pipeline isn't added to records sent to Kinesis. | *none* | | **Time Key Format** | `time_key_format` | `strftime`-compliant format string for the timestamp. This option is used with `time_key`. | `%Y-%m-%dT%H:%M:%S` | | **Log Key** | `log_key` | By default, the entire log record is sent to Kinesis. If you specify a key name with this option, only the value of that key is sent to Kinesis. For example, if you're using the Fluentd Docker log driver, you can specify `log_key log`, and only the log message is sent to Kinesis. | *none* | | **Enable Auto Retry Requests** | `auto_retry_requests` | Whether to immediately retry failed requests to AWS services one time. This option doesn't affect the normal Telemetry Pipeline retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient or random networking issues. Accepted values: `true`, `false`. | `false` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Amazon Kinesis Streams destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/aws-streams Configure the Amazon Kinesis Streams destination plugin in Chronosphere Telemetry Pipeline to send logs to Amazon Kinesis Streams. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Amazon Kinesis Streams [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `kinesis_streams`) lets you stream and analyze your data by sending it directly to your Amazon Kinesis stream. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ----------------------- | -------- | ---------------------------------------------------- | ----------- | | **Region** | `region` | Required. The AWS region to which to send your data. | `us-east-1` | | **Kinesis Stream Name** | `stream` | Kinesis Stream name. | *none* | ### AWS Authentication | Name | Key | Description | Default | | -------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------- | | **IAM Role ARN** | `role_arn` | Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role to assume, for uses like cross-account access. | *none* | | **Kinesis Streams API Endpoint** | `endpoint` | Custom endpoint for the Kinesis API. | *none* | | **STS API Endpoint** | `sts_endpoint` | Custom endpoint for the STS API. | *none* | | **External ID for STS API** | `external_id` | Specifies an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID. | *none* | ### Advanced | Name | Key | Description | Default | | ------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | | **Time Key** | `time_key` | Adds the timestamp to the record under this key. By default, the timestamp from Telemetry Pipeline isn't added to records sent to Kinesis. | *none* | | **Time Key Format** | `time_key_format` | `strftime`-compliant format string for the timestamp. This option is used with `time_key`. | `%Y-%m-%dT%H:%M:%S` | | **Log Key** | `log_key` | By default, the entire log record is sent to Kinesis. If you specify a key name with this option, only the value of that key is sent to Kinesis. For example, if you're using the Fluentd Docker log driver, you can specify `log_key log`, and only the log message is sent to Kinesis. | *none* | | **Enable Auto Retry Requests** | `auto_retry_requests` | Whether to immediately retry failed requests to AWS services one time. This option doesn't affect the normal Telemetry Pipeline retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient or random networking issues. Accepted values: `true`, `false`. | `false` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Axiom destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/axiom Configure the Axiom destination plugin in Chronosphere Telemetry Pipeline to send logs to Axiom. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Axiom [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `Axiom`) lets you integrate your telemetry data with Axiom for additional processing and analysis. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ---------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | | **Axiom Token** | `header` | Adds an HTTP header key-value pair. You can set multiple `header` values. | *none* | | **Dataset Name** | `uri` | Required. The dataset name used when routing data to the Axiom service. For more information, see [Send data from Fluent Bit to Axiom](https://axiom.co/docs/send-data/fluent-bit) in the Axiom documentation. | `/api/v1/datasets/$DATASET_NAME/ingest` | ### Advanced | Name | Key | Description | Default | | -------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | **Host** | `host` | Required. IP address or hostname of Axiom data collection endpoint. | `cloud.axiom.co` | | **Port** | `port` | Required. The port used to communicate with the Axiom service. Use port `443` for TLS. | `443` | | **Compress** | `compress` | Sets the payload compression mechanism. | `gzip` | | **HTTP Proxy** | `proxy` | Specifies an HTTP Proxy. The expected format of this value is `http://host:port`. | *none* | | **Format** | `format` | Required. Specifies the data format to be used in the HTTP request body. Accepted values: `json`, `json_lines`, `json_stream`, `msgpack`. | `json` | | **JSON Date Format** | `json_date_format` | Specifies the format of the date. Accepted values: `double`, `epoch`, `iso8601` (for example, `2018-05-30T09:39:52.000681Z`), `java_sql_timestamp` (for example, `2018-05-30 09:39:52.000681`, which can be used with AWS Athena). | `iso8601` | | **JSON Date Key** | `json_date_key` | Specifies the name of the date field in output. | `_time` | | **Body Key** | `body_key` | Specifies the key that contains the body. | *none* | | **Header Tag** | `header_tag` | Sets an HTTP header whose value is the tag of the record. | *none* | | **Header Key** | `headers_key` | Specifies the key that contains the headers. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | # Azure Blob Storage destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/azure-blob Configure the Azure Blob Storage destination plugin in Chronosphere Telemetry Pipeline to send logs to Azure Blob Storage. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Azure Blob Storage [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `azure_blob`) lets you store your logs, metrics, and other data types directly in your Azure Blob Storage account. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ------------------------------ | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Azure Storage Account Name** | `account_name` | Required. Azure Storage account name. | *none* | | **Azure Storage Shared Key** | `shared_key` | Required. Specifies the Azure Storage Shared Key to authenticate against the service. | *none* | | **Container Name** | `container_name` | Required. The name of the container that will contain the blobs. | *none* | | **Path** | `path` | Optional path to store your blobs. If your blob name is `myblob`, you can specify sub-directories where to store it using path. Setting a path to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. | *none* | | **Blob Type** | `blob_type` | Specifies the blob type. Accepted values: `appendblob`, `blockblob`. | `appendblob` | | **Auto Create Container** | `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. Accepted values: `on`, `off`. | `on` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Azure Event Hubs destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/azure-eventhub Configure the Azure Event Hubs destination plugin in Chronosphere Telemetry Pipeline to send logs to Azure Event Hubs. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Azure Event Hubs [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `kafka`, alias: `Azure_Event_Hub`) lets you configure your telemetry pipeline to stream your logs and metrics data directly to Azure Event Hubs. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------- | ------- | | **Event Hub Namespace** | `brokers` | Required. The Event Hubs namespace that can be found within the configuration settings. | *none* | | **Event Hub Topic** | `topics` | Required. The Event Hubs topic to write information to. | *none* | | **Connection String Key** | `rdkafka.sasl.password` | Required. The Event Hubs connection string from within the Connection Access Policy set for the source. | *none* | ### Advanced | Name | Key | Description | Default | | ---------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | | **Message Key** | `message_key` | Optional key to store the message. | *none* | | **Message Key Field** | `message_key_field` | If set, this value indicates the message key. If not set or found in the record, `message_key` will be used (if set). | *none* | | **Timestamp Key** | `timestamp_key` | Sets the key to store the record timestamp. | *none* | | **Timestamp Format** | `timestamp_format` | Sets the timestamp format. Accepted values: `double`, `iso8601`. | `double` | | **Body Key** | `body_key` | Specifies the key that contains the body. | *none* | | **Queue Full Retries** | `queue_full_retries` | Telemetry Pipeline queues data into rdkafka library, if for some reason the underlying library can't flush the records the queue might fills up blocking new addition of records. The `queue_full_retries` option set the number of local retries to enqueue the data. The default value is 10 times, the interval between each retry is 1 second. Setting the `queue_full_retries` value to 0 sets an unlimited number of retries. | `10` | | **SASL Username** | `rdkafka.sasl.username` | SASL username. | `$ConnectionString` | | **Security Protocol** | `rdkafka.security.protocol` | This is the security protocol for Azure Event Hub. If you require OAuth 2.0 or OpenID authentication, open a support ticket prior to changing this setting. | `SASL_SSL` | | **SASL Mechanism** | `rdkafka.sasl.mechanism` | This is the transport mechanism for the SASL connection. | `PLAIN` | ### Extended librdkafka parameters This plugin uses the [librdkafka](https://github.com/confluentinc/librdkafka) library. Certain configuration parameters available through the Telemetry Pipeline UI are based on librdkafka settings. These parameters generally use the `rdkafka.` prefix. In addition to the parameters available through the Telemetry Pipeline UI, you can customize any of the [librdkafka configuration properties](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) by adding them to a pipeline configuration file. To do so, append the `rdkafka.` prefix to the name of that property. For example, to customize the `socket.keepalive.enable` property, add the `rdkafka.socket.keepalive.enable` key to your configuration file. # Azure Data Explorer destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/azure-explorer Configure the Azure Data Explorer destination plugin in Chronosphere Telemetry Pipeline to send logs to Azure Data Explorer. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Azure Data Explorer [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `azure_kusto`) lets you configure your telemetry pipeline to send your logs and metrics data directly to Azure Data Explorer. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ---------------------- | -------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------ | | **Tenant ID** | `tenant_id` | Required. The tenant or domain ID of the Azure AD registered app. | *none* | | **Client ID** | `client_id` | Required. The client ID of the Azure AD registered app. | *none* | | **Client Secret** | `client_secret` | Required. The client secret of the Azure AD registered app (App Secret). | *none* | | **Ingestion Endpoint** | `ingestion_endpoint` | Required. The clusters ingestion endpoint. | `https://ingest-cluster_name.region.kusto.windows.net` | | **Database Name** | `database_name` | Required. The database name. | *none* | | **Table Name** | `table_name` | Required. The table name. | *none* | ### Advanced | Name | Key | Description | Default | | ------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ----------- | | **Ingestion Mapping Reference** | `ingestion_mapping_reference` | The name of a JSON ingestion mapping that will be used to map the ingested payload into the table columns. | *none* | | **Log Key** | `log_key` | Key name of the log content. | `log` | | **Enable Tag Key** | `include_tag_key` | If `true`, the tag is appended to output. The key name is used `tag_key` property. Accepted values: `true`, `false`. | `true` | | **Tag Key** | `tag_key` | Specifies the key name where the tag is stored. | `tag` | | **Enable Time Key** | `include_time_key` | If `true`, a timestamp is appended to output. The key name is used `time_key` property. Accepted values: `true`, `false`. | `true` | | **Time Key** | `time_key` | Specifies the key name where the timestamp is stored. | `timestamp` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Azure Logs Ingestion API destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/azure-ingestion Configure the Azure Logs Ingestion API destination plugin in Chronosphere Telemetry Pipeline to send logs to Azure Logs Ingestion API. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Azure Logs Ingestion API [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `azure_logs_ingestion`) lets you configure your telemetry pipeline to output data to Azure Monitor using Azure's Log Ingestion API. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ----------------- | --------------- | ------------------------------------------------------------------------------------------- | ------- | | **Tenant ID** | `tenant_id` | Required. The tenant ID of the Azure AD app. | *none* | | **Client ID** | `client_id` | Required. The client ID of the Azure AD app. | *none* | | **Client Secret** | `client_secret` | Required. The client secret of the Azure AD app (App Secret). | *none* | | **DCE URL** | `dce_url` | Required. Data Collection Endpoint (DCE) URL. | *none* | | **DCR ID** | `dcr_id` | Required. Data Collection Rule (DCR) immutable ID. | *none* | | **Table Name** | `table_name` | Required. The name of the custom log table (include the \_CL suffix as well if applicable). | *none* | ### Advanced | Name | Key | Description | Default | | ------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Time Key** | `time_key` | Specifies the key name where the timestamp will be stored. | *none* | | **Enable Time Generated** | `time_generated` | If `true`, the HTTP request header `time-generated-field` is included so Azure can override the timestamp with the key specified by `time_key` option. Accepted values: `true`, `false`. | `true` | | **Enable Compression** | `compress` | If `true`, enable HTTP payload gzip compression. Accepted values: `true`, `false`. | `true` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Azure Monitor Logs destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/azure-monitor Configure the Azure Monitor Logs destination plugin in Chronosphere Telemetry Pipeline to send logs to Azure Monitor. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Azure Monitor Logs [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `azure`) lets you configure your telemetry pipeline to send your logs and metrics data directly to Azure Monitor Logs. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ----------------------------- | ------------- | ----------------------------------------------------------------------------------- | ------- | | **Customer / Workspace ID** | `customer_id` | Required. Customer ID or WorkspaceID string. | *none* | | **Client Authentication Key** | `shared_key` | Required. The primary or the secondary Connected Sources client authentication key. | *none* | ### Advanced | Name | Key | Description | Default | | ------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Event Type Name** | `log_type` | The name of the event type. | `fluentbit` | | **Time Key** | `time_key` | Specifies the key name where the timestamp is stored. | `@timestamp` | | **Enable Time Generated** | `time_generated` | If `true`, the HTTP request header `time-generated-field` is included so Azure can override the timestamp with the key specified by `time_key` option. Accepted values: `true`, `false`. | `false` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Azure Sentinel destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/azure-sentinel Configure the Azure Sentinel destination plugin in Chronosphere Telemetry Pipeline to send logs to Azure Sentinel. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Azure Sentinel [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `azure`, alias: `Azure_Sentinel`) lets you configure your telemetry pipeline to send security-related logs and events to Azure Sentinel. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ----------------------------- | ------------- | ----------------------------------------------------------------------------------- | ------- | | **Customer / Workspace ID** | `customer_id` | Required. Customer ID or WorkspaceID string. | *none* | | **Client Authentication Key** | `shared_key` | Required. The primary or the secondary Connected Sources client authentication key. | *none* | ### Advanced | Name | Key | Description | Default | | ------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Event Type Name** | `log_type` | The name of the event type. | `fluentbit` | | **Time Key** | `time_key` | Optional parameter to specify the key name where the timestamp is stored. | `@timestamp` | | **Enable Time Generated** | `time_generated` | If `true`, the HTTP request header `time-generated-field` is included so Azure can override the timestamp with the key specified by the `time_key` option. Accepted values: `true`, `false`. | `false` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Chronosphere Logs destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/chronosphere Configure the Chronosphere Logs destination plugin in Chronosphere Telemetry Pipeline to send logs to Chronosphere. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Chronosphere Logs [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `Chronosphere`) lets you configure your telemetry pipeline to output log data to Chronosphere Observability Platform. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | --------- | -------- | ------------------------------------------------------------------ | ---------------------------- | | **Host** | `host` | Required. Your Observability Platform tenant. | *`TENANT`*`.chronosphere.io` | | **Port** | `port` | Required. The port to communicate with Observability Platform. | `443` | | **Key** | `header` | Required. The header type to include with your request. | `API-Token` | | **Value** | `header` | Required. Your ingest API token to authenticate with Chronosphere. | *none* | * * ### Advanced | Name | Key | Description | Default | | -------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------- | | **URI** | `uri` | Required. HTTP URI for the web server. For example, `/api/v1/data/logs/ingest` for JSON format. | `/api/v1/data/logs/ingest` | | **Format** | `format` | The data format to use in the HTTP request. | `json` | | **Compress** | `compress` | Sets payload compression. Accepted values: `gzip`, *none*. | `gzip` | | **HTTP Proxy** | `proxy` | Specifies an HTTP proxy. The expected format is `http://host:port`. | *none* | | **JSON Date Format** | `json_date_format` | Specifies the date format. Accepted values: `double`, `epoch`, `iso8601` (such as `2018-05-30T09:39:52.000681Z`), `java_sql_timestamp` (such as `2018-05-30 09:39:52.000681`). | `iso8601` | | **JSON Date Key** | `json_date_key` | Specifies the name of the date field in output. | `_time` | | **Body Key** | `body_key` | Specifies the key that contains the body. | *none* | | **Header Tag** | `header_tag` | Sets an HTTP header whose value is the tag of the record. | *none* | | **Header Key** | `headers_key` | Specifies the key that contains the headers. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | ## Example The following YAML configuration defines a Fluent Bit input and a Chronosphere Logs output. ```yaml theme={null} pipeline: inputs: - name: forward alias: Fluent_Bit port: "5170" tls.verify: on tls.debug: "1" tag: forward.cb3bae48-0000-4a51-0000-f6ccd3f0ed0c filters: - name: lua match: forward.cb3bae48-0000-4a51-0000-f6ccd3f0ed0c active: "false" outputs: - name: http alias: Chronosphere host: TENANT.chronosphere.io port: "443" header: - API-Token f31b...ed7t292 uri: /api/v1/data/logs/ingest format: json compress: gzip json_date_format: iso8601 json_date_key: _time tls: true tls.verify: on tls.debug: "1" net.keepalive: true net.keepalive_idle_timeout: 30s net.connect_timeout: 10s net.connect_timeout_log_error: true net.keepalive_max_recycle: "2000" core.metadata: name: Chronosphere Logs ingest Match_Regex: .{0,} ``` # Clickhouse destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/clickhouse Configure the Clickhouse destination plugin in Chronosphere Telemetry Pipeline to send logs to Clickhouse. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Clickhouse [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `Clickhouse`) lets you send your telemetry pipeline data to Clickhouse. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------- | ------ | -------------------------------------------------------------- | ------- | | **Host** | `host` | Required. IP address or hostname of the Clickhouse cloud host. | *none* | | **Port** | `port` | Required. TCP host port of the target HTTP server. | `8443` | ### Advanced | Name | Key | Description | Default | | -------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | **URI** | `uri` | Required. Specifies a HTTP URI for the target web server. | `/?query=INSERT+INTO+fluentbit.kube+FORMAT+JSONEachRow` | | **Format** | `format` | Required. Specifies the data format to use in the HTTP request body. Accepted values: `json`, `json_lines`, `json_stream`. | `json_stream` | | **JSON Date Format** | `json_date_format` | Specifies the date format. Accepted values: `double`, `epoch`, `iso8601`, `java_sql_timestamp`. | `epoch` | | **JSON Date Key** | `json_date_key` | Specifies the name of the date field in output. | `timestamp` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | # Confluent Cloud DataSet destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/confluent-cloud Configure the Confluent Cloud DataSet destination plugin in Chronosphere Telemetry Pipeline to send logs to Confluent Cloud. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Confluent Cloud [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `kafka`, alias: `confluent_cloud_output`) lets you send your telemetry pipeline data to Confluent Cloud. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------- | --------------------------------- | | **Confluent Cloud Bootstrap Servers** | `brokers` | Required. The Confluent Cloud bootstrap servers can be found within the cluster configuration settings. | `SERVERNAME.confluent.cloud:9092` | | **Confluent Cloud Topic** | `topics` | Required. The Confluent Cloud Topic to send information to. | *none* | | **Confluent Cloud API Key** | `rdkafka.sasl.username` | Required. Confluent Cloud API Key. | *none* | | **Confluent Cloud API Secret** | `rdkafka.sasl.password` | Required. The Confluent Cloud API Secret. | *none* | | **Format** | `format` | Specifies a data format. Accepted values: `json`, `msgpack`. | `json` | ### Advanced | Name | Key | Description | Default | | ---------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | **Message Key** | `message_key` | Optional key to store the message. | *none* | | **Message Key Field** | `message_key_field` | If set, the value of `Message_Key_Field` in the record will indicate the message key. If not set nor found in the record, `Message_Key` will be used if set. | *none* | | **Timestamp Key** | `timestamp_key` | Sets the key to store the record timestamp. | *none* | | **Timestamp Format** | `timestamp_format` | Sets the format. Accepted values: `double`, `iso8601`. | `double` | | **Body Key** | `body_key` | Specifies the key that contains the body. | *none* | | **Queue Full Retries** | `queue_full_retries` | Telemetry Pipeline queues data into `rdkafka` library. If the underlying library can't flush the records the queue might fill and block new addition of records. This option sets the number of local retries to enqueue the data. The interval between each retry is 1 second. Setting `queue_full_retries` to `0` to set an unlimited number of retries. | `10` | | **Security Protocol** | `rdkafka.security.protocol` | The security protocol used to communicate with Confluent Cloud. | `SASL_SSL` | | **SASL Mechanism** | `rdkafka.sasl.mechanism` | The SASL authentication mechanism for the API. | `PLAIN` | ### Extended librdkafka parameters This plugin uses the [librdkafka](https://github.com/confluentinc/librdkafka) library. Certain configuration parameters available through the Telemetry Pipeline UI are based on librdkafka settings. These parameters generally use the `rdkafka.` prefix. In addition to the parameters available through the Telemetry Pipeline UI, you can customize any of the [librdkafka configuration properties](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) by adding them to a pipeline configuration file. To do so, append the `rdkafka.` prefix to the name of that property. For example, to customize the `socket.keepalive.enable` property, add the `rdkafka.socket.keepalive.enable` key to your configuration file. # Coralogix destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/coralogix Configure the Coralogix destination plugin in Chronosphere Telemetry Pipeline to send logs to Coralogix. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Coralogix [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `Coralogix`) lets you configure your telemetry pipeline to transmit your data to Coralogix for further processing and analysis. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------------------------- | -------- | ------------------------------------------------------------ | ------------------- | | **Coralogix API Endpoint** | `host` | Required. The API endpoint for Coralogix. | `api.coralogix.com` | | **Coralogix Private Key** | `header` | Add your Coralogix Private Key. You can enter multiple keys. | *none* | ### Advanced | Name | Key | Description | Default | | -------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | **URI** | `uri` | Required. Coralogix URI. | `/logs/rest/singles` | | **Compress** | `compress` | Sets the payload compression mechanism. Accepted values: `gzip`, *none*. | `gzip` | | **HTTP Proxy** | `proxy` | Specifies an HTTP Proxy. The expected format of this value is `http://host:port`. | *none* | | **Format** | `format` | Specifies the data format to be used in the HTTP request body. Accepted values: `json`, `json_lines`, `json_stream`, `msgpack`. | `json_lines` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | # Cortex XSIAM destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/cortex-xsiam Configure the Cortex XSIAM destination plugin in Chronosphere Telemetry Pipeline to send logs to Cortex XSIAM. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Cortex XSIAM [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `paloaltocortex`) lets you configure your telemetry pipeline to send logs to Palo Alto Networks Cortex XSIAM. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | | Cortex URL | `host` | Required. Specifies your Cortex XSIAM webhook URL. | `api-{tenant external URL}.crtx.us.paloaltonetworks.com` | | Port | `port` | Required. Specifies TCP port of the target HTTP server. | `443` | | Cortex URI and Instance | `uri` | Required. Specifies the URI to use as part of the request. | `/xsoar/instance/execute/my_instance_01` | | Headers | `header` | Required. Sets an HTTP header key-value pair. Use your ingest API token to authenticate with Cortex XSIAM. | `Authorization [REPLACE WITH TOKEN]` | ### Advanced | Name | Key | Description | Default | | ---------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------- | --------- | | Format | `format` | Specifies the data format to use in the HTTP request body. Accepted values: `json_lines`, `json`, `json_stream`. | `json` | | Compress | `compress` | Sets the payload compression mechanism. Accepted values: `gzip`, *none*. | *none* | | HTTP Proxy | `proxy` | Specifies an HTTP proxy. The expected format of this value is `http://host:port`. | *none* | | JSON Date Format | `json_date_format` | Sets the date format. Accepted values: `double`, `epoch`, `iso8601`, `java_sql_timestamp`. | `iso8601` | | JSON Date Key | `json_date_key` | Specifies the name of the date field in output. | `_time` | | Body Key | `body_key` | Specifies the key that contains the format. | *none* | | Header Tag | `header_tag` | Sets an HTTP header whose value is the tag of the record. | *none* | | Header Key | `headers_key` | Specifies the key that contains the headers. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `true` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | # CrowdStrike destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/crowdstrike Configure the CrowdStrike destination plugin in Chronosphere Telemetry Pipeline to send logs to CrowdStrike. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The CrowdStrike [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `Humio`) lets you configure your telemetry pipeline to send data from any of your sources to the CrowdStrike LogScale platform. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------- | -------- | ----------------------------------------------------------------------------------- | --------------------------- | | **Host** | `host` | Required. IP address or hostname of the LogScale Server. | `cloud.community.humio.com` | | **Port** | `port` | Required. TCP port of the targeted HTTP server. | `443` | | **Headers** | `header` | Add the LogScale Ingest Token after the `Bearer`. You can add more than one header. | *none* | ### Advanced | Name | Key | Description | Default | | -------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | | **URI** | `uri` | Required. Specifies an optional HTTP URI for the target web server. | `/api/v1/ingest/json` | | **Format** | `format` | Required. Specifies the data format to be used in the HTTP request body. Accepted values: `json`, `json_lines`, `json_stream`. | `json` | | **Compress** | `compress` | Sets the payload compression mechanism. Accepted values: `gzip`, *none*. | *none* | | **HTTP Proxy** | `proxy` | Specifies an HTTP Proxy. The expected format of this value is `http://host:port`. | *none* | | **JSON Date Format** | `json_date_format` | Specifies the format of the date. Accepted values: `double`, `epoch`, `iso8601` (for example, `2018-05-30T09:39:52.000681Z`), `java_sql_timestamp` (for example, `2018-05-30 09:39:52.000681`, which can be used with AWS Athena). | `iso8601` | | **JSON Date Key** | `json_date_key` | Specifies the name of the date field in output. | `_time` | | **Body Key** | `body_key` | Specifies the key that contains the body. | *none* | | **Header Tag** | `header_tag` | Sets an HTTP header whose value is the tag of the record. | *none* | | **Header Key** | `headers_key` | Specifies the key that contains the headers. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Datadog Logs destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/datadog Configure the Datadog Logs destination plugin in Chronosphere Telemetry Pipeline to send logs to Datadog. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Datadog Logs [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `datadog`) lets you configure your telemetry pipeline to integrate your telemetry data with Datadog Logs, letting you store, analyze, and visualize your log data using logging tools in Datadog. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ------------------- | -------- | -------------------------------------------------------- | ------- | | **Datadog API Key** | `apikey` | Required. Datadog API Key found within account settings. | *none* | ### Advanced | Name | Key | Description | Default | | ------------------------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | **Datadog Server** | `host` | The Datadog server where you're sending your logs. | `http-intake.logs.datadoghq.com` | | **Datadog Service Name** | `dd_service` | The human-readable name for your service generating the logs. The name of your app or database. | *none* | | **Datadog Source Name** | `dd_source` | A human readable name for the underlying technology of your service. For example, `postgres` or `nginx`. | *none* | | **Datadog Tags** | `dd_tags` | The tags you want to assign to your logs in Datadog. | *none* | | **Datadog Message Key** | `dd_message_key` | By default, the plugin searches for the key 'log' and remap the value to the key `message`. If the property is set, the plugin searches the property name key. | *none* | | **Include Tag Key** | `include_tag_key` | If `true`, tag is appended to output. The key name is used `tag_key` property. Accepted values: `true`, `false`. | `false` | | **Tag Key** | `tag_key` | The key name of tag. If `include_tag_key` is false, This property is ignored. | `tagkey` | | **JSON Date Key** | `json_date_key` | Date key name for output in log. | `timestamp` | | **Compression** | `compress` | Compresses the payload in GZIP format, Datadog supports and recommends setting this to 'gzip'. | `gzip` | | **Proxy** | `proxy` | Specifies an HTTP Proxy. The expected format of this value is `http://host:port`. HTTPS isn't supported. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Devo Cloud destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/devo Configure the Devo Cloud destination plugin in Chronosphere Telemetry Pipeline to send logs to Devo Cloud. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Devo Cloud [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `syslog`, alias: `devo_cloud`) lets you send your telemetry pipeline data to Devo Cloud. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | | **Devo Collector** | `host` | Required. The URL of the Devo Collector. | `collector-us.devo.com` | | **Message Key** | `syslog_message_key` | Required, or the message is empty. The key name from the original record that contains the message to deliver. | *none* | | **Devo Tag Key** | `syslog_appname_key` | The key within the message to be used as the Devo Tag. If **Devo Tag Preset** is specified then this field is not required. | *none* | | **Devo Tag Preset** | `syslog_appname_preset` | The preset Devo Tag. This field is overwritten if **Devo Tag Key** is set and a key of a record is matched. | `my.app.calyptia.core` | | **CA Certificate File Path** | `tls.ca_file` | CA certificate file found in Devo Cloud under **Administration > Relays and ELBs > Syslog Certificates > Chain CA**. After downloading, upload as a file and reference as `{{files.}}`. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Path to the private key file, found in Devo Cloud under **Administration > Relays and ELBs > Syslog Certificates > Private Key**. Download, then upload as a file and reference as `{{files.}}`. | *none* | | **Private Key File Password** | `tls.key_passwd` | The private key password displayed in Devo Cloud under **Administration > Relays and ELBs > Syslog Certificates > Private Key**. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | **Port** | `port` | Required. The HTTPS port of the Devo Cloud Relay. | `443` | | **Mode** | `mode` | Required. Desired transport type. Available options: `tcp`, `tls`, and `udp`. | `tcp` | | **RFC Format** | `syslog_format` | Required. The format used to send data to Devo Cloud. `rfc3164` is the default and formatting must be changed for `rfc5424`. | `rfc3164` | | **Syslog Severity Key** | `syslog_severity_key` | The key name from the original record that contains the Syslog severity number. | *none* | | **Syslog Severity Preset** | `syslog_severity_preset` | The preset severity number. Overwritten if `syslog_severity_key` is set and a key of a record matches. | `informational` | | **Syslog Facility Key** | `syslog_facility_key` | The key name from the original record that contains the Syslog facility number. | *none* | | **Syslog Facility Preset** | `syslog_facility_preset` | The preset facility number. Overwritten if `syslog_facility_key` is set and a key of a record matches. | `local7` | | **Syslog Hostname Key** | `syslog_hostname_key` | The key name from the original record that contains the hostname that generated the message. | *none* | | **Syslog Hostname Preset** | `syslog_hostname_preset` | The preset hostname. Overwritten if `syslog_hostname_key` is set and a key of a record matches. | `calyptia` | | **Syslog ProcID Key** | `syslog_procid_key` | The key name from the original record that contains the Process ID that generated the message. | *none* | | **Syslog ProcID Preset** | `syslog_procid_preset` | The preset process ID. Overwritten if `syslog_procid_key` is set and a key of a record is matched. | *none* | | **Syslog Message ID Key** | `syslog_msgid_key` | The key name from the original record that contains the Message ID associated to the message. | *none* | | **Syslog Message ID Preset** | `syslog_msgid_preset` | The preset message ID.Overwritten if `syslog_msgid_key` is set and a key of a record is matched. | *none* | | **Syslog Structured Data (SD) Key** | `syslog_sd_key` | The key name from the original record that contains the Structured Data (SD) content. | *none* | | **Syslog Maxsize** | `syslog_maxsize` | The maximum size allowed per message. The value must be an integer representing the number of bytes allowed. If no value is provided, the default size is set depending of the protocol version specified by `syslog_format`. RFC3164 sets max size to 1024 bytes. RFC5424 sets the size to 2048 bytes. | `2048` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ------------ | ------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | Enables or disables TLS/SSL support. | `true` | | **TLS Certificate Validation** | `tls.verify` | Turns TLS/SSL certificate validation on or off. TLS must be on for this setting to be enabled. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Dynatrace Logs destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/dynatrace Configure the Dynatrace Logs destination plugin in Chronosphere Telemetry Pipeline to send logs to Dynatrace. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Dynatrace Logs [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `Dynatrace`) lets you configure your telemetry pipeline to send your log data to Dynatrace. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------- | -------------------------- | | **Dynatrace Environment URL** | `host` | Required. The URL for your Dynatrace instance. For example, `{YOUR-ENVIRONMENT_ID}.live.dynatrace.com`. | *none* | | **Dynatrace API Token** | `header` | Adds an HTTP header key-value pair for the API token. You can set multiple headers. | *See the following table.* | The following headers are to set your Dynatrace API Token: | Key | Value | | ----------------- | ---------------------------------------------- | | **Authorization** | `Api-Token [REPLACE WITH DYNATRACE API TOKEN]` | | **Content-Type** | `application/json; charset=utf-8` | ### Advanced | Name | Key | Description | Default | | ---------------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | | **Port for Dynatrace** | `port` | The port number of the Dynatrace Logs API. | `443` | | **URI** | `uri` | The URI of the Dynatrace Logs API. | `/api/v2/logs/ingest` | | **Allow Duplicated Headers** | `allow_duplicated_headers` | Setting to ensure no duplicate Authorization API-Tokens are present. | `false` | | **HTTP Proxy** | `proxy` | Specifies an HTTP Proxy. The expected format of this value is `http://host:port`. | *none* | | **Format** | `format` | Specifies the data format to be used in the HTTP request body. Accepted values: `json`, `json_lines`, `json_stream`. | `json` | | **JSON Date Format** | `json_date_format` | Specifies the format of the date. Accepted values: `double`, `epoch`, `iso8601` (for example, `2018-05-30T09:39:52.000681Z`), `java_sql_timestamp` (for example, `2018-05-30 09:39:52.000681`, which can be used with AWS Athena). | `iso8601` | | **JSON Date Key** | `json_date_key` | Specifies the name of the date field in output. | `timestamp` | | **Body Key** | `body_key` | Specifies the key that contains the body. | *none* | | **Header Tag** | `header_tag` | Sets an HTTP header whose value is the tag of the record. | *none* | | **Header Key** | `headers_key` | Specifies the key that contains the headers. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Elasticsearch destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/elasticsearch Configure the Elasticsearch destination plugin in Chronosphere Telemetry Pipeline to send logs to Elasticsearch. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Elasticsearch [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `es`) lets you configure your telemetry pipeline to store, search, and visualize your data using Elasticsearch's indexing and querying capabilities. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ------------------- | ----------------- | ------------------------------------------------------------------------ | --------------- | | **Host** | `host` | Required. IP address or hostname of the target Elasticsearch instance. | `127.0.0.1` | | **Port** | `port` | Required. TCP port of the target Elasticsearch instance. | `9200` | | **Index** | `index` | Required. Index name. | `calyptia-core` | | **Logstash Format** | `logstash_format` | Enables Logstash format compatibility. Accepted values: `true`, `false`. | `false` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced | Name | Key | Description | Default | | ------------------------ | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Replace Dots** | `replace_dots` | When enabled, replaces field name dots with underscore, required by versions of Elasticsearch. Accepted values: `true`, `false`. | `false` | | **Type** | `type` | Required. Type Name. | `_doc` | | **Suppress Type Name** | `suppress_type_name` | If true, mapping types are removed. For v7.0.0 or later. Accepted values: `true`, `false`. | `false` | | **Buffer Size** | `buffer_size` | Specifies the buffer size used to read the response from the Elasticsearch HTTP service. Use this option for debugging purposes where it's required to read full responses. Response size grows depending of the number of records inserted. To set an unlimited amount of memory set this value to `false`, otherwise the value must be according to the Unit Size specification. | `512k` | | **Path** | `path` | Elasticsearch accepts new data on HTTP query path `/_bulk`. It's also possible to serve Elasticsearch behind a reverse proxy on a sub path. This option defines such path on the Telemetry Pipeline side. It adds only a path prefix in the indexing HTTP POST URI. | *none* | | **Pipeline** | `pipeline` | Newer versions of Elasticsearch let you set up filters, also called pipelines. This option lets you define which pipeline the database should use. For performance reasons, it's strongly suggested to do parsing and filtering on Telemetry Pipeline side. | *none* | | **Generate ID** | `generate_id` | When enabled, generates `_id` for outgoing records. This prevents duplicate records when retrying ES. Accepted values: `true`, `false`. | `false` | | **Write Operation** | `write_operation` | Operation to use to write in bulk requests. | `create` | | **ID Key** | `id_key` | If set, `_id` is the value of the key from incoming record. | *none* | | **Replace Dots** | `current_time_index` | Uses current time for index generation instead of message record. Accepted values: `true`, `false`. | `false` | | **Logstash Prefix** | `logstash_prefix` | When `logstash_format` is enabled, the Index name is composed using a prefix and the date. For example, if `logstash_prefix` is equal to `mydata`, your index becomes `mydata-YYYY.MM.DD`. The last string appended belongs to the date when the data is being generated. | *none* | | **Logstash Prefix Key** | `logstash_prefix_key` | When included, the value in the record that belongs to the key will be looked up and over-write the `logstash_prefix` for index generation. If the key-value isn't found in the record, the `logstash_prefix` option acts as a fallback. Nested keys are supported through record accessor pattern. | *none* | | **Logstash Date Format** | `logstash_dateformat` | Time format (based on `strftime`) to generate the second part of the Index name. | `%Y.%m.%d` | | **Time Key** | `time_key` | When `logstash_format` is enabled, each record will get a new timestamp field. The `time_key` property defines the name of that field. | `@timestamp` | | **Time Key Format** | `time_key_format` | When `logstash_format` is enabled, this property defines the format of the timestamp. | *none* | | **Time Key Nanos** | `time_key_nanos` | When `logstash_format` is enabled, enabling this property sends nanosecond precision timestamps. Accepted values: `true`, `false`. | `false` | | **Include Tag Key** | `include_tag_key` | When enabled, it append the Tag name to the record. Accepted values: `true`, `false`. | `false` | | **Tag Key** | `tag_key` | When `include_tag_key` is enabled, this property defines the key name of the tag in the message. | `flb-key` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | ### AWS Authentication | Name | Key | Description | Default | | ----------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------- | ------- | | **Enable AWS Authentication** | `aws_auth` | Enable AWS Sigv4 Authentication. Accepted values: `true`, `false`. | `false` | | **AWS Region** | `aws_region` | AWS Region of your service. | *none* | | **AWS STS Endpoint** | `aws_sts_endpoint` | Custom endpoint for the AWS STS API, used with the `aws_role_arn` option. | *none* | | **AWS IAM Assume Role** | `aws_role_arn` | Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role to assume, for uses like cross-account access. | *none* | | **AWS External ID** | `aws_external_id` | Specifies an external ID for the STS API, can be used with the `aws_role_arn` parameter if your role requires an external ID. | *none* | ### Elastic Cloud Authentication | Name | Key | Description | Default | | -------------------------------------------- | ------------ | ---------------------------------------------- | ------- | | **Elastic Cloud ID** | `cloud_id` | Elastic Cloud ID of the cluster to connect to. | *none* | | **Elastic Cloud Authentication Credentials** | `cloud_auth` | Elastic Cloud authentication credentials. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Debugging | Name | Key | Description | Default | | ---------------- | -------------- | -------------------------------------------------------------------------------------------------------------------- | ------- | | **Trace Output** | `trace_output` | When enabled, print the Elasticsearch API calls to `stdout`. For diagnostics only. Accepted values: `true`, `false`. | `false` | | **Trace Error** | `trace_error` | When enabled, print the Elasticsearch exception to `stderr`. For diagnostics only. Accepted values: `true`, `false`. | `false` | # Exabeam destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/exabeam Configure the Exabeam destination plugin in Chronosphere Telemetry Pipeline to send logs to Exabeam. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Exabeam [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `tcp`, alias: `Exabeam`) lets you configure your telemetry pipeline to output data to Exabeam. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ---------- | -------- | ------------------------------------------------------------------------------------------------------------------- | ----------- | | **Host** | `host` | Required. IP address or hostname of the target service. | `127.0.0.1` | | **Port** | `port` | Required. TCP port of the target service. | `80` | | **Format** | `format` | Required. Specifies the data format to be printed. Accepted values: `json`, `json_lines`, `json_stream`, `msgpack`. | `msgpack` | ### Advanced | Name | Key | Description | Default | | -------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------- | ------- | | **JSON Date Format** | `json_date_format` | Specifies the format of the date. Accepted values: `double`, `epoch`, `iso8601`, `java_sql_timestamp`. | *none* | | **JSON Date Key** | `json_date_key` | Specifies the name of the time key in the output record. To disable the time key set the value to `false`. | *none* | | **Raw Message Key** | `raw_message_key` | Specifies a key to contains the message to send in raw format. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Forward destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/forward Configure the Forward destination plugin in Chronosphere Telemetry Pipeline to send logs, metrics, and traces over Forward. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Forward [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `forward`) lets you configure your telemetry pipeline to send processed telemetry data to your Forward Services. This plugin supports both TCP and UDP transport protocols and provides options for configuring data buffering and batching to optimize network usage. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :--------------------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | | **Host** | `host` | Required. Target host where Fluent-Bit or Fluentd are listening for Forward messages. | `127.0.0.1` | | **Port** | `port` | Required. TCP port of the target service. | `24224` | | **Timestamps as Integers** | `time_as_integer` | Sets timestamps in integer format, and enables compatibility mode for Fluentd v0.12 series. Accepted values: `true`, `false`. | `false` | | **Upstream Configuration Path** | `upstream` | If Forward will connect to an Upstream instead of a simple host, this property defines the absolute path for the Upstream configuration file. For more details, refer to the Upstream Servers documentation section. | *none* | | **Unix Socket** | `unix_path` | Specifies the path to the Unix socket to send a Forward message. If set, Upstream is ignored. | *none* | | **Tag** | `tag` | Overwrites the transmitted tag, which lets the receiving pipeline either start fresh or to attribute the source. | *none* | | **Send Options** | `send_options` | Always send options, with `size` equaling the count of messages. Accepted values: `true`, `false`. | `false` | | **Require Ack Response** | `require_ack_response` | Indicates whether to send `chunk`-option and wait for `ack` response from server. Enables at-least-once and receiving server can control rate of traffic. Requires Fluentd v0.14.0 or later server. Accepted values: `true`, `false`. | `false` | | **Compress** | `compress` | Enables gzip compression. Incompatible with `time_as_integer=true` and tags set dynamically using the Rewrite Tag filter. Requires Fluentd v0.14.7 or later server. Accepted values: `gzip`, *none*. | *none* | | **Workers** | `workers` | Enables one or more dedicated threads for this output. | `2` (`0` for versions 1.8.12 and earlier) | # Google Cloud Operations Suite destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/gcp-ops Send pipeline data to Google Cloud Operations Suite. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Google Cloud Operations Suite [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `stackdriver`) lets you configure your telemetry pipeline to stream your log data and metrics to Google Cloud Operations Suite. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### GCP Authentication | Name | Key | Description | Default | | ----------------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------- | | **Google Service Credentials Path** | `google_service_credentials` | The Service Credentials file lets Telemetry Pipeline communicate directly with Google Cloud Services. For information about how to set up service credentials, see the [Google Cloud Observability documentation](https://cloud.google.com/logging/docs/agent/logging/authorization#create-service-account). | *none* | | **GCP Service Account Email** | `service_account_email` | Account email associated to the service. Available only if no credentials file is provided. | *none* | | **GCP Service Account Secret** | `service_account_secret` | Private key content associated with the service account. Available only if no credentials file is provided. | *none* | | **Metadata Server** | `metadata_server` | Private key content associated with the service account. Available only if no credentials file is provided. | `http://metadata.google.internal` | ### Advanced | Name | Key | Description | Default | | ---------------------------------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Region Location** | `location` | The Google Cloud or AWS region in which to store data about the resource. If the resource type is either `generic_node` or `generic_task`, this field is required. | *none* | | **Namespace Identifier** | `namespace` | A namespace identifier, such as a cluster name or environment. If the resource type is either `generic_node` or `generic_task`, this field is required. | *none* | | **Node ID Identifier** | `node_id` | A unique identifier for the node within the namespace, such as hostname or IP address. If the resource type is `generic_node`, this field is required. | *none* | | **Job Identifier** | `job` | An identifier for a grouping of related task, such as the name of a microservice or distributed batch. If the resource type is `generic_task`, this field is required. | *none* | | **Task ID Identifier** | `task_id` | A unique identifier for the task within the namespace and job, such as a replica index identifying the task within the job. If the resource type is `generic_task`, this field is required. | *none* | | **GCP Project ID** | `export_to_project_id` | The Google Cloud project to receive these logs. Defaults to the project ID of the `google_service_credentials` file, or the `project_id` from Google's `metadata.google.internal` server. | *none* | | **Resource Type** | `resource` | Sets the resource type of data. Accepted values: `k8s_container`, `k8s_node`, `k8s_pod`, `global`, `generic_node`, `generic_task`, `gce_instance`. | *none* | | **Kubernetes Cluster Name** | `k8s_cluster_name` | The name of the cluster that the container (node or pod based on the resource type) is running in. If the resource type is `k8s_container`, `k8s_node`, or `k8s_pod`, this field is required. | *none* | | **Kubernetes Cluster Location** | `k8s_cluster_location` | The physical location of the cluster that contains (node or pod based on the resource type) the container. If the resource type is `k8s_container`, `k8s_node`, or `k8s_pod`, this field is required. | *none* | | **Labels Key** | `labels_key` | The value of this field is used by the Stackdriver output plugin to find the related labels from `jsonPayload` and then extract the value of it to set the LogEntry labels. | *none* | | **Labels** | `labels` | Optional list of comma-separated of strings specifying key-value pairs. The resulting labels are combined with the elements in obtained from `labels_key` to set the LogEntry labels. Elements from labels override duplicate values from `labels_key`. | *none* | | **Log Name Key** | `log_name_key` | The value of this field is used by the plugin to extract `logName` from `jsonPayload` and set the `logName` field. | *none* | | **Tag Prefix** | `tag_prefix` | Sets the `tag_prefix` used to validate the tag of logs with k8s resource type. Without this option, the tag of the log must be in format of `k8s_container(pod/node).*` to use the `k8s_container` resource type. Now the tag prefix is configurable by this option, being aware of the ending dot. | *none* | | **Severity Key** | `severity_key` | Specifies the name of the key from the original record that contains the severity information. | *none* | | **Enable Cloud Trace Integration** | `autoformat_stackdriver_trace` | Rewrite the trace field to include the projectID and format it for use with Cloud Trace. When enabled, the user can get the correct result by printing only the traceID (usually 32 characters). Accepted values: `true`, `false`. | `false` | | **Custom K8s Regex** | `custom_k8s_regex` | Sets a custom regular expression to extract field like `pod_name`, `namespace_name`, `container_name`, and `docker_id` from the `local_resource_id` in logs. This is helpful if the value of pod or node name contains dots. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Google Cloud BigQuery destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/google-bigquery Configure the Google Cloud BigQuery destination plugin in Chronosphere Telemetry Pipeline to send logs to Google Cloud BigQuery. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Google Cloud BigQuery [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `bigquery`) lets you configure your telemetry pipeline to send your log data and metrics to Google Cloud BigQuery. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Google Service Credentials Path** | `google_service_credentials` | Required. The Service Credentials file lets Telemetry Pipeline communicate directly with Google Cloud Services. For information about how to set up service credentials, see the [Google Cloud Observability documentation](https://cloud.google.com/logging/docs/agent/logging/authorization#create-service-account). | *none* | | **Google Project Id** | `project_id` | The project ID containing the BigQuery dataset to stream into. If the service file is provided then the project ID is taken from there. | *none* | | **Existing Data Set ID** | `dataset_id` | The dataset ID of the BigQuery dataset to write into. This dataset must exist in your project. | *none* | | **Existing Table ID** | `table_id` | The table ID of the BigQuery table to write into. This table must exist in the specified dataset and the schema must match the output. | *none* | ### Advanced | Name | Key | Description | Default | | --------------------------------------- | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Skip Invalid Rows** | `skip_invalid_rows` | If `true`, inserts all valid rows of a request, even if invalid rows exist. If `false`, causes the entire request to fail if any invalid rows exist. Accepted values: `true`, `false`. | `false` | | **Ignore Unknown Values** | `ignore_unknown_values` | Accepts rows that contain values that don't match the schema, and ignores unknown values. When set to `false`, treats unknown values as errors. Accepted values: `true`, `false`. | `false` | | **Enable Workload Identity Federation** | `enable_workload_identity_federation` | Enables workload identity federation as an alternative authentication method. Can't be used with service account credentials file or environment variable. AWS is the only supported identity provider. Accepted values: `true`, `false`. | `false` | | **AWS Region for BigQuery** | `aws_region` | Google Cloud region for BigQuery. | *none* | | **GCP Project Number** | `project_number` | Google Cloud project number where the identity provider was created. Used to construct the full resource name of the identity provider. | *none* | | **GCP Pool Id** | `pool_id` | Google Cloud workload identity pool where the identity provider was created. Used to construct the full resource name of the identity provider. | *none* | | **GCP Provider Id** | `provider_id` | Google Cloud workload identity provider. Used to construct the full resource name of the identity provider. Only AWS accounts are supported. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Google Chronicle destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/google-chronicle Configure the Google Chronicle destination plugin in Chronosphere Telemetry Pipeline to send logs to Google Chronicle. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Google Chronicle [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `chronicle`) lets you configure your telemetry pipeline to output data to Google Chronicle. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Chronicle Settings | Name | Key | Description | Default | | ----------------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | | **Google Service Credentials Path** | `google_service_credentials` | The Service Credentials file lets Telemetry Pipeline communicate directly with Google Cloud Services using a [service account](https://cloud.google.com/logging/docs/agent/logging/authorization#create-service-account). | *none* | | **GCP Service Account Email** | `service_account_email` | Account email associated with the service. Available only if no credentials file has been provided. | *none* | | **GCP Service Account Secret** | `service_account_secret` | Private key content associated with the service account. Available only if no credentials file has been provided. | *none* | | **GCP Project Id** | `project_id` | The project ID containing the tenant of Google Chronicle to stream into. | *none* | | **Google Chronicle Customer ID** | `customer_id` | Required. The customer ID to identify the tenant of Google Chronicle to stream into. | *none* | | **Google Chronicle Log Type** | `log_type` | Required. The log type to handle the request entries. Users must set up the valid, [supported log type](https://cloud.google.com/chronicle/docs/ingestion/parser-list/supported-default-parsers) or the Google Chronicle service denies log ingestion. | *none* | | **Google Chronicle Log Key** | `log_key` | Required. Specifies the field that contains the log type specified in `log_type`. By default, the whole log record is sent to Google Chronicle. | *none* | ### Advanced | Name | Key | Description | Default | | ------------------- | -------- | ----------------------------------------------------------------------------------------------------------- | ------- | | **Region Location** | `region` | The GCP region in which to store Google Chronicle security logs. Accepted values: `ASIA`, `EU`, `UK`, `US`. | `US` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Grafana Loki destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/grafana-loki Configure the Grafana Loki destination plugin in Chronosphere Telemetry Pipeline to send logs to Grafana Loki. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Grafana Loki [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `loki`) lets you configure your telemetry pipeline to send your log data to Grafana Loki. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | -------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------- | ----------- | | **Host** | `host` | Required. Loki hostname or IP address. Don't include the sub path, such as `loki/api/v1/push`. Include only the base hostname or URL. | `127.0.0.1` | | **Port** | `port` | Required. TCP port of the target Loki service. | `3100` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | ### Advanced | Name | Key | Description | Default | | --------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | **Tenant ID** | `tenant_id` | Tenant ID used by default to push logs to Loki. If omitted or empty, it's assumed that Loki is running in single-tenant mode and no `X-Scope-OrgID` header is sent. | *none* | | **Tenant ID Key** | `tenant_id_key` | Required. If set, `X-Scope-OrgID` will be the value of the key from incoming record. It can be useful to set `X-Scode-OrgID` dynamically. | *none* | | **Labels** | `labels` | Labels for API requests. | `job=fluentbit` | | **Label Keys** | `label_keys` | Comma-separated list of keys to use as stream labels. | *none* | | **Enable Auto Kubernetes Labels** | `auto_kubernetes_labels` | If set to true, adds all Kubernetes labels to Loki labels. Accepted values: `true`, `false`. | `false` | | **Drop Single Key** | `drop_single_key` | If set to true and only a single key remains, the log line sent to Loki is the value of that key. Accepted values: `true`, `false`. | `false` | | **Remove Keys** | `remove_keys` | Comma-separated list of keys to remove. | *none* | | **Line Format** | `line_format` | Format to use when flattening the record to a log line. Accepted values: `json` (sent log line is the Telemetry Pipeline record dumped as JSON), `key_value` (log line is each item in the record concatenated together, separated by a single space, in the format `=`). | `json` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Graylog destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/graylog Configure the Graylog destination plugin in Chronosphere Telemetry Pipeline to send logs to Graylog. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Graylog [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `gelf`) lets you send your telemetry pipeline data to Graylog. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | --------------- | ------------- | ---------------------------------------------------------------------------------------------- | ----------- | | **Host** | `Host` | IP address or hostname of the target Graylog server. | `127.0.0.1` | | **Port** | `Port` | The port that your Graylog GELF input is listening on. | `12201` | | **Mode** | `Mode` | The protocol to use (`tls`, `tcp` or `udp`). | `udp` | | **Packet Size** | `Packet_Size` | If `mode` is `udp`, sets the size of packets to be sent. | `1420` | | **Compress** | `Compress` | If `true`, and `mode` is `udp`, compresses your UDP packets. Accepted values: `true`, `false`. | `true` | ### GELF settings | Name | Key | Description | Default | | -------------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------- | --------------- | | **GELF Short Message Key** | `Gelf_Short_Message_Key` | A short descriptive message (must be set in GELF). | `short_message` | | **GELF Timestamp Key** | `Gelf_Timestamp_Key` | Your log timestamp (should be set in GELF). | `timestamp` | | **GELF Host Key** | `Gelf_Host_Key` | This key value is used as the name of the host, source, or app that sent this message. (must be set in GELF). | `host` | | **GELF Full Message Key** | `Gelf_Full_Message_Key` | Key to use as the long message. (Optional in GELF). | *none* | | **GELF Level Key** | `Gelf_Level_Key` | Key to be used as the log level. Its value must be in standard syslog levels between 0 and 7. (Optional in GELF). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # HTTP destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/http Configure the HTTP destination plugin in Chronosphere Telemetry Pipeline to send logs over HTTP. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The HTTP [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `HTTP`) lets you configure your telemetry pipeline to send your data to remote servers or services over the internet to an HTTP endpoint, such as a web server or an API. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ---------- | -------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------ | | **Host** | `host` | Required. IP address or hostname of the target HTTP Server. | `0.0.0.0` | | **Port** | `port` | Required. TCP port of the target HTTP Server. | `80` | | **URI** | `uri` | Required. Specifies an optional HTTP URI for the target web server. | `/` | | **Format** | `format` | Required. Specifies the data format to be used in the HTTP request body. Accepted values: `json`, `json_lines`, `json_stream`. | `json_lines` | ### Advanced | Name | Key | Description | Default | | -------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Headers** | `header` | Adds an HTTP header key-value pair. You can set multiple headers. | *none* | | **Compress** | `compress` | Sets the payload compression mechanism. Accepted values: `gzip`, *no value* | *none* | | **HTTP Proxy** | `proxy` | Specifies an HTTP Proxy. The expected format of this value is `http://host:port`. | *none* | | **JSON Date Format** | `json_date_format` | Specifies the format of the date. Accepted values: `double`, `epoch`, `iso8601` (for example, `2018-05-30T09:39:52.000681Z`), `java_sql_timestamp` (for example, `2018-05-30 09:39:52.000681`, which can be used with AWS Athena). | *none* | | **JSON Date Key** | `json_date_key` | Specifies the name of the date field in output. | *none* | | **Body Key** | `body_key` | Specifies the key that contains the body. | *none* | | **Header Tag** | `header_tag` | Sets an HTTP header whose value is the tag of the record. | *none* | | **Header Key** | `headers_key` | Specifies the key that contains the headers. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | ### AWS Authentication | Name | Key | Description | Default | | -------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------- | ------- | | **Enable AWS Authentication** | `aws_auth` | Enable AWS Sigv4 Authentication. Accepted values: `true`, `false`. | `false` | | **AWS Destination Service Code** | `aws_service` | AWS destination service code, used by SigV4 authentication. | *none* | | **AWS Region** | `aws_region` | AWS Region of your service. | *none* | | **AWS STS Endpoint** | `aws_sts_endpoint` | Custom endpoint for the AWS STS API, used with the `aws_role_arn` option. | *none* | | **AWS IAM Assume Role** | `aws_role_arn` | Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role to assume, for uses like cross-account access. | *none* | | **AWS External ID** | `aws_external_id` | Specifies an external ID for the STS API. Can be used with the `aws_role_arn` parameter if your role requires an external ID. | *none* | # InfluxDB destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/influxdb Configure the InfluxDB destination plugin in Chronosphere Telemetry Pipeline to send logs and metrics to InfluxDB. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The InfluxDB [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `influxdb`) lets you configure your telemetry pipeline to route data to your InfluxDB instances. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------ | ---------- | ----------------------------------------------------------------------------------------------------------------- | -------------- | | **Host** | `host` | IP address or hostname of the target HTTP Server. | `127.0.0.1` | | **Port** | `port` | TCP port of the target HTTP Server. | `80` | | **Influx v1 - Database** | `database` | InfluxDB Database Name where records are inserted. If you're using a newer version of Influx, use Bucket instead. | `calyptiacore` | | **Influx v2 - Bucket** | `bucket` | The amount of time between each API call. | *none* | | **Influx v2 - Org** | `org` | InfluxDB Organization name where the bucket is (v2 only). | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | -------------- | ----------------------------------------------------------------------------------- | ------- | | **Headers** | `http_header` | Adds an HTTP header key-value pair. Multiple headers can be set. | *none* | | **Sequence Tag** | `sequence_tag` | The name of the tag whose value is incremented for consecutive simultaneous events. | `_seq` | | **Custom URI Endpoint** | `uri` | Custom URI endpoint. | *none* | | **Enable Auto Tags** | `auto_tags` | Custom URI endpoint. Accepted values: `true`, `false`. | `false` | | **Tag Keys** | `tag_keys` | Space-separated list of keys that need to be tagged. | *none* | ### Basic Authentication | Name | Key | Description | Default | | -------------------------- | ------------- | -------------------------------------------------------------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | | **Influx v2 - HTTP Token** | `http_token` | Authentication token used with InfluDB v2. If specified, both `http_user` and `http_passwd` are ignored. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Kafka destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/kafka Configure the Kafka destination plugin in Chronosphere Telemetry Pipeline to send logs to Kafka. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Kafka [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `kafka`) lets you configure your telemetry pipeline to integrate your data with Apache Kafka for real-time processing and analysis. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | **Brokers** | `brokers` | Required. Single of multiple list of Kafka brokers. For example, `192.168.1.3:9092` or `192.168.1.4:9092`. | *none* | | **Topics** | `topics` | Required. Single entry or list of comma-separated topics that Telemetry Pipeline uses to send messages to Kafka. If only one topic is set, that topic is used for all records. If multiple topics exists, the one set in the record by `topic_key` is used. | `calyptia-core` | | **Format** | `format` | Specifies data format. Accepted values: `json`, `msgpack`. | `json` | ### Advanced | Name | Key | Description | Default | | ---------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | **Message Key** | `message_key` | Optional key to store the message. | *none* | | **Message Key Field** | `message_key_field` | If set, the value of `message_key_field` in the record will indicate the message key. If not set or found in the record, `message_key` will be used (if set). | *none* | | **Timestamp Key** | `timestamp_key` | Sets the key to store the record timestamp. | *none* | | **Timestamp Format** | `timestamp_format` | Sets the format of the timestamp. Accepted values: `double`, `iso8601`. | `double` | | **Body Key** | `body_key` | Specifies the key that contains the body. | *none* | | **Queue Full Retries** | `queue_full_retries` | Telemetry Pipeline queues data into rdkafka library. If the underlying library can't flush the records, the queue might fill up, blocking new addition of records. This key sets the number of local retries to enqueue the data. The interval between each retry is one second. Setting the key to `0` sets an unlimited number of retries. | `10` | ### Extended librdkafka parameters This plugin uses the [librdkafka](https://github.com/confluentinc/librdkafka) library. Certain configuration parameters available through the Telemetry Pipeline UI are based on librdkafka settings. These parameters generally use the `rdkafka.` prefix. In addition to the parameters available through the Telemetry Pipeline UI, you can customize any of the [librdkafka configuration properties](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) by adding them to a pipeline configuration file. To do so, append the `rdkafka.` prefix to the name of that property. For example, to customize the `socket.keepalive.enable` property, add the `rdkafka.socket.keepalive.enable` key to your configuration file. # New Relic Logs destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/new-relic Configure the New Relic Logs destination plugin in Chronosphere Telemetry Pipeline to send logs to New Relic. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The New Relic Logs [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `nrlogs`) lets you configure your telemetry pipeline to send your log data to New Relic Logs. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | --------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **New Relic API Key** | `api_key` | Required. Your key for data ingestion. The API key is also called the ingestion key. For more details about how to generate the API key, see the [official documentation](https://docs.newrelic.com/docs/apis/get-started/intro-apis/types-new-relic-api-keys#event-insert-key). From a configuration perspective, either an `api_key` or an `license_key` is required. New Relic suggests using `api_key`. | *none* | ### Advanced | Name | Key | Description | Default | | ------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | | **New Relic License Key** | `license_key` | Optional authentication parameter for data ingestion. New Relic suggests you instead use the `api_key`. [License Key information](https://docs.newrelic.com/docs/accounts/install-new-relic/account-setup/license-key). | *none* | | **Compression** | `compress` | Compression type for the payload. | `gzip` | | **New Relic Base URI** | `base_uri` | Full address of New Relic API endpoint. For the EU endpoint, set this key to `https://log-api.eu.newrelic.com/log/v1`. | `https://log-api.newrelic.com/log/v1` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Null destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/null Configure the Null destination plugin in Chronosphere Telemetry Pipeline for logs, metrics, and traces. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. Null is a type of data destination that lets you discard your telemetry data without writing it to any external storage or output destination. The Null [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `"null"`) is a tool for testing or debugging purposes when you want to simulate a data flow without actually writing data to disk or sending it to an external system. With the Null destination plugin, you can configure your pipeline to discard any telemetry data that passes through it, ensuring that the data isn't saved or transmitted to any external system. This can be used for verifying the configuration of your pipeline or for troubleshooting issues in your data processing workflow. The Null destination plugin can help you avoid unnecessary data storage or network traffic. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :--------------------------: | | | | | # Observe destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/observe Configure the Observe destination plugin in Chronosphere Telemetry Pipeline to send logs to Observe. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Observe [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `Observe`) lets you configure your telemetry pipeline to send your data to Observe through HTTPS. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------------------- | -------- | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- | | **Observe Customer Id** | `host` | Required. IP address or hostname of Observes data collection endpoint. `$(OBSERVE_CUSTOMER_ID)` is your Customer ID. | `OBSERVE_CUSTOMER_ID.collect.observeinc.com` | | **Headers** | `header` | Adds an HTTP header key-value pair. Multiple headers can be set. | *See the following table.* | The following headers are to set your connection: | Key | Value | | --------------------- | ------------------------------------- | | **Authorization** | `Bearer (REPLACE WITH OBSERVE TOKEN)` | | **X-Observe-Decoder** | `fluent` | ### Advanced | Name | Key | Description | Default | | -------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | **URI** | `uri` | Required. The default URI used for the Observe service. | `/v1/http/fluentbit` | | **Port** | `port` | Required. The port used to communicate with the Observe service. | `443` | | **Compress** | `compress` | Sets the payload compression mechanism. Accepted values: `gzip`, *none*. | `gzip` | | **HTTP Proxy** | `proxy` | Specifies an HTTP Proxy. The expected format of this value is `http://host:port`. | *none* | | **Format** | `format` | Required. Specifies the data format to be used in the HTTP request body. Accepted values: `json`, `json_lines`, `json_stream`, `msgpack`. | `msgpack` | | **JSON Date Format** | `json_date_format` | Specifies the format of the date. Accepted values: `double`, `epoch`, `iso8601` (for example, `2018-05-30T09:39:52.000681Z`), `java_sql_timestamp` (for example, `2018-05-30 09:39:52.000681`, which can be used with AWS Athena). | *none* | | **JSON Date Key** | `json_date_key` | Specifies the name of the date field in output. | *none* | | **Body Key** | `body_key` | Specifies the key that contains the body. | *none* | | **Header Tag** | `header_tag` | Sets an HTTP header whose value is the tag of the record. | *none* | | **Header Key** | `headers_key` | Specifies the key that contains the headers. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | # OpenSearch destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/opensearch Configure the OpenSearch destination plugin in Chronosphere Telemetry Pipeline to send logs to OpenSearch. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The OpenSearch [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `opensearch`) lets you configure your telemetry pipeline to send your data to OpenSearch. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | --------- | ------- | --------------------------------------------------------- | --------------- | | **Host** | `host` | IP address or hostname of the target OpenSearch instance. | `127.0.0.1` | | **Port** | `port` | TCP port of the target OpenSearch instance. | `9200` | | **Index** | `index` | Index name. | `calyptia-core` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced | Name | Key | Description | Default | | ------------------------ | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Replace Dots** | `replace_dots` | When enabled, replaces field name dots with underscore, required by versions of OpenSearch. Accepted values: `true`, `false`. | `false` | | **Type** | `type` | Type Name. | `_doc` | | **Suppress Type Name** | `suppress_type_name` | If true, mapping types are removed. For v7.0.0 or later. Accepted values: `true`, `false`. | `true` | | **Buffer Size** | `buffer_size` | Specifies the buffer size used to read the response from the OpenSearch HTTP service. This option is useful for debugging purposes where is required to read full responses. Response size grows depending of the number of records inserted. To set an unlimited amount of memory set this value to `false`, otherwise the value must be according to the Unit Size specification | `512k` | | **Path** | `path` | OpenSearch accepts new data on HTTP query path `/_bulk`. It's also possible to serve Elasticsearch behind a reverse proxy on a sub path. This option defines such path on the Telemetry Pipeline side. It adds only a path prefix in the indexing HTTP POST URI. | *none* | | **Generate ID** | `generate_id` | When enabled, generates `_id` for outgoing records. This prevents duplicate records when retrying ES. Accepted values: `true`, `false`. | *false* | | **Write Operation** | `write_operation` | Operation to use to write in bulk requests. | `create` | | **ID Key** | `id_key` | If set, `_id` will be the value of the key from incoming records. | *none* | | **Replace Dots** | `current_time_index` | Uses the current time for index generation instead of the message record. Accepted values: `true`, `false`. | `false` | | **Logstash Prefix** | `logstash_prefix` | When `logstash_format` is enabled, the index name is composed using a prefix and the date. For example, if `logstash_prefix` is `mydata`, your index becomes `mydata-YYYY.MM.DD`. The last string appended belongs to the date when the data is being generated. | *none* | | **Logstash Prefix Key** | `logstash_prefix_key` | When included, the value in the record that belongs to the key will be looked up and over-write the `logstash_prefix` for index generation. If the key-value isn't found in the record, the `logstash_prefix` option acts as a fallback. Nested keys are supported through record accessor patterns. | *none* | | **Logstash Date Format** | `logstash_dateformat` | Time format (based on `strftime`) to generate the second part of the index name. | `%Y.%m.%d` | | **Time Key** | `time_key` | When `logstash_format` is enabled, each record gets a new timestamp field. The `time_key` property defines the name of that field. | `@timestamp` | | **Time Key** | `time_key_format` | When `logstash_format` is enabled, this property defines the format of the timestamp. | *none* | | **Time Key Nanos** | `time_key_nanos` | When `logstash_format` is enabled, enabling this property sends nanosecond precision timestamps. Accepted values: `true`, `false`. | `false` | | **Include Tag Key** | `inclulde_tag_key` | When enabled, it appends the tag name to the record. Accepted values: `true`, `false`. | `false` | | **Tag Key** | `tag_key` | When `inclulde_tag_key` is enabled, this property defines the key name of the tag in the message. | `flb-key` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | ### AWS Authentication | Name | Key | Description | Default | | ----------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------- | ------- | | **Enable AWS Authentication** | `aws_auth` | Enable AWS Sigv4 authentication. Accepted values: `true`, `false`. | `false` | | **AWS Region** | `aws_region` | AWS region of your service. | *none* | | **AWS STS Endpoint** | `aws_sts_endpoint` | Custom endpoint for the AWS STS API, used with the `aws_role_arn` option. | *none* | | **AWS IAM Assume Role** | `aws_role_arn` | Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role to assume, for uses like cross-account access. | *none* | | **AWS External ID** | `aws_external_id` | Specifies an external ID for the STS API. Can be used with the `aws_role_arn` parameter if your role requires an external ID. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Debugging | Name | Key | Description | Default | | ---------------- | -------------- | ------------------------------------------------------------------------------------------------------------------ | ------- | | **Trace Output** | `trace_output` | When enabled, print the OpenSearch API calls to `stdout`. For diagnostics only. Accepted values: `true`, `false`. | `false` | | **Trace Error** | `trace_error` | When enabled, print the OpenSearch exceptions to `stderr`. For diagnostics only. Accepted values: `true`, `false`. | `false` | # OpenTelemetry Output destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/opentelemetry Configure the OpenTelemetry Output destination plugin in Chronosphere Telemetry Pipeline to send logs, metrics, and traces to OpenTelemetry. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The OpenTelemetry [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `opentelemetry`, alias: `OpenTelemetry`) lets you configure your telemetry pipeline to send telemetry data to a wide range of data sources, such as observability platforms, analytics tools, and logging systems. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :--------------------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | --------------- | ------------- | ----------------------------------------------------------------------------------------------------------- | ----------- | | **Host** | `host` | Required. IP address or hostname of the target HTTP Server. | `127.0.0.1` | | **Port** | `port` | Required. TCP port of the target HTTP Server. | `80` | | **Metrics URI** | `metrics_uri` | Specifies an optional HTTP URI for the target web server listening for metrics. For example, `/v1/metrics`. | `/` | | **Logs URI** | `logs_uri` | Specifies an optional HTTP URI for the target web server listening for metrics. For example, `/v1/logs`. | `/` | | **Traces URI** | `traces_uri` | Specifies an optional HTTP URI for the target web server listening for metrics. For example, `/v1/traces`. | `/` | ### Advanced | Name | Key | Description | Default | | --------------------------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------- | ------- | | **Headers** | `header` | Adds an HTTP header key-value pair. You can set multiple headers. | *none* | | **Additional Prometheus Metric Labels** | `add_label` | Adds custom labels to all metrics exposed through the OpenTelemetry exporter. You can set multiple headers. | *none* | | **HTTP Proxy** | `proxy` | Specifies an HTTP proxy. The expected format of this value is `http://host:port`. | *none* | | **Log Response Payload** | `log_response_payload` | Logs the response payload within the Core Pipeline log. | `false` | | **Compress** | `compress` | Sets the payload compression mechanism. Accepted values: `gzip`, *none*. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | # Oracle Log Analytics destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/oracle Configure the Oracle Log Analytics destination plugin in Chronosphere Telemetry Pipeline to send logs to Oracle. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Oracle Log Analytics [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `oracle_log_analytics`) lets you configure your telemetry pipeline to output log data to Oracle Log Analytics. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | **Configuration File Location** | `config_file_location` | Required. The location of the configuration file containing OCI authentication details. See [SDK and CLI Configuration File](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm#SDK_and_CLI_Configuration_File) for steps on generating the configuration file. | `~/.oci/config` | | **Profile Name** | `profile_name` | Required. OCI Config Profile Name to be used from the configuration file. | `DEFAULT` | | **Namespace** | `namespace` | Required. OCI Tenancy Namespace in which the collected log data is to be uploaded. | *none* | | **Proxy** | `proxy` | Defines a proxy. Must be in the format `http://host:port`. Supports the HTTP protocol only. | *none* | ### Advanced | Name | Key | Description | Default | | ------------------------ | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **OCI Config in Record** | `oci_config_in_record` | If set to `true`, the following `oci_la_*` parameters are read from the record itself instead of the output plugin configuration. Accepted values: `true`, `false`. | `false` | | **OCI Log Group ID** | `oci_la_log_group_id` | Required. The OCID of the Logging Analytics Log Group where the logs must be stored. | *none* | | **OCI Log Source Name** | `oci_la_log_source_name` | Required. The Logging Analytics Source that must be used to process the log records. | *none* | | **OCI Entity ID** | `oci_la_entity_id` | The ID of the Logging Analytics Entity. | *none* | | **OCI Entity Type** | `oci_la_entity_type` | The entity type of the Logging Analytics Entity. | *none* | | **OCI Log Path** | `oci_la_log_path` | Specifies the original location of the log files. | *none* | | **OCI Global Metadata** | `oci_la_global_metadata` | Specifies additional global metadata along with original log content to Logging Analytics. The format is `key value`. This option can be set multiple times. | *none* | | **OCI Metadata** | `oci_la_metadata` | Specifies additional metadata for a log event along with original log content to Logging Analytics. The format is `key value`. This option can be set multiple times. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ## Example The following YAML configuration defines a Fluent Bit input and an Oracle Log Analytics output. ```yaml theme={null} pipeline: inputs: - Name: forward alias: Fluent_Bit port: "5170" tls.verify: on tls.debug: "1" tag: forward.0dd7a9e7-ed46-4c31-bbd3-40e8b607f501 outputs: - Name: oracle_log_analytics config_file_location: ~/.oci/config profile_name: DEFAULT namespace: NAMESPACE proxy: http://host:port oci_la_log_group_id: log_group_id oci_la_log_source_name: REST API oci_la_entity_id: entity oci_la_entity_type: entity_type oci_la_log_path: log_path oci_la_global_metadata: - key1 value1 oci_la_metadata: - key2 value2 tls: true tls.verify: on tls.debug: "1" net.keepalive: true net.keepalive_idle_timeout: 30s net.connect_timeout: 10s net.connect_timeout_log_error: true net.keepalive_max_recycle: "2000" Match_Regex: .{0,} ``` # Prometheus Exporter destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/prometheus-exporter Configure the Prometheus Exporter destination plugin in Chronosphere Telemetry Pipeline to expose metrics for scraping. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Prometheus Exporter [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `prometheus_exporter`) lets you export data using the Prometheus Exporter. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :-----------------: | :--------------------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------------- | ----------- | -------------------------------------------------------------------------------------------------------- | --------- | | **Host** | `host` | Required. The address Telemetry Pipeline binds to when hosting Prometheus metrics. | `0.0.0.0` | | **Port** | `port` | Required. The port Telemetry Pipeline binds to when hosting Prometheus metrics. | `2021` | | **Add Labels** | `add_label` | Adds custom labels to all metrics exposed through the Prometheus Exporter. You can have multiple fields. | `Value` | # Prometheus Remote Write destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/prometheus-remote-write Configure the Prometheus Remote Write destination plugin in Chronosphere Telemetry Pipeline to send metrics over Prometheus Remote Write. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Prometheus Remote Write [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `prometheus_remote_write`, alias: `PrometheusRemoteWrite`) lets you configure your telemetry pipeline to send metrics data in a standardized format over HTTP to a configured remote endpoint. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :-----------------: | :--------------------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------- | ------ | ------------------------------------------------------------------- | ----------- | | **Host** | `host` | Required. IP address or hostname of the target HTTP Server. | `127.0.0.1` | | **Port** | `port` | Required. TCP port of the target HTTP Server. | `80` | | **URI** | `uri` | Required. Specifies an optional HTTP URI for the target web server. | `/` | If you configured [Chronosphere Collector](/ingest/metrics-traces/collector) to serve as a Prometheus remote write backend, view the Chronosphere Collector configuration for the configured host, port, and URL. ### Advanced | Name | Key | Description | Default | | --------------------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------- | ------- | | **Headers** | `header` | Adds an HTTP header key-value pair. You can set multiple headers. | *none* | | **Additional Prometheus Metric Labels** | `add_label` | Adds custom labels to all metrics exposed through the Prometheus exporter. You can set multiple headers. | *none* | | **HTTP Proxy** | `proxy` | Specifies an HTTP Proxy. The expected format of this value is `http://host:port`. | *none* | | **Log Response Payload** | `log_response_payload` | Logs the response payload within the Core Pipeline log. | `false` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | # Redpanda Cloud destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/redpanda Configure the Redpanda Cloud destination plugin in Chronosphere Telemetry Pipeline to send logs to Redpanda Cloud. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Redpanda Cloud [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `kafka`, alias: `RedPandaCloud`) lets you send your telemetry pipeline data to the Redpanda Cloud service. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | **Brokers** | `brokers` | Required. List of Kafka Brokers. For example, `192.168.1.3:9092, 192.168.1.4:9092` | *none* | | **Topics** | `topics` | Required. Single entry or comma-separated list of topics that Telemetry Pipeline uses to send messages to Kafka. If only one topic is set, that one will be used for all records. If multiple topics exist, the topic set in the record by `Topic_Key` is used. | `calyptia-core` | | **Format** | `format` | Required. Specifies a data format. Accepted values: `json`, `msgpack`. | `json` | ### Advanced | Name | Key | Description | Default | | ---------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | **Security Protocol** | `rdkafka.security.protocol` | The Security Protocol used to communicate with Redpanda. | `sasl_ssl` | | **SASL Mechanism** | `rdkafka.sasl.mechanisms` | The SASL authentication mechanism for the API. | `SCRAM-SHA-256` | | **Redpanda Username** | `rdkafka.sasl.username` | Your Redpanda username. | *none* | | **Redpanda Password** | `rdkafka.sasl.password` | Required. Password for your Redpanda username. | *none* | | **Message Key** | `message_key` | Optional key to store the message. | *none* | | **Message Key Field** | `message_key_field` | If set, the value of `Message_Key_Field` in the record will indicate the message key. If not set nor found in the record, `Message_Key` will be used if set. | *none* | | **Timestamp Key** | `timestamp_key` | Sets the key to store the record timestamp. | *none* | | **Timestamp Format** | `timestamp_format` | Sets the format to `iso8601` or `double`. | `double` | | **Body Key** | `body_key` | Specifies the key that contains the body. | *none* | | **Queue Full Retries** | `queue_full_retries` | Telemetry Pipeline queues data into `rdkafka` library. If the underlying library can't flush the records the queue might fill and block new addition of records. This option sets the number of local retries to enqueue the data. The interval between each retry is 1 second. Setting `queue_full_retries` to `0` to set an unlimited number of retries. | `10` | ### Extended librdkafka parameters This plugin uses the [librdkafka](https://github.com/confluentinc/librdkafka) library. Certain configuration parameters available through the Telemetry Pipeline UI are based on librdkafka settings. These parameters generally use the `rdkafka.` prefix. In addition to the parameters available through the Telemetry Pipeline UI, you can customize any of the [librdkafka configuration properties](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) by adding them to a pipeline configuration file. To do so, append the `rdkafka.` prefix to the name of that property. For example, to customize the `socket.keepalive.enable` property, add the `rdkafka.socket.keepalive.enable` key to your configuration file. # Scalyr DataSet destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/scalyr Configure the Scalyr DataSet destination plugin in Chronosphere Telemetry Pipeline to send logs to Scalyr. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Scalyr [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `scalyr_dataset`) lets you send your telemetry pipeline data to Scalyr as Scalyr datasets. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------------- | ----------------------- | -------------------------------------------------------------- | ------- | | **Host** | `host` | Required. IP address or hostname of the target Scalyr server. | *none* | | **Port** | `port` | Required. TCP host port of the target Scalyr server. | `0` | | **DataSet Token** | `scalyr_datasets_token` | Specifies the Authentication Bearer Token for the DataSet API. | *none* | ### Advanced | Name | Key | Description | Default | | --------------------------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Compress** | `compress` | Sets the payload compression mechanism. | *none* | | **HTTP Buffer Size** | `http_buffer_size` | Specifies the buffer size used to read the response from the `scalyr_dataset` HTTP service. | *none* | | **Enable HTTP Debug Bad Request** | `http_debug_bad_request` | If `true`, and if the HTTP server response status code is `400 Bad Request`, it will print the full HTTP request and response to the `stdout` interface. This feature is available for debugging purposes. Accepted values: `true`, `false`. | `false` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # Slack destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/slack Configure the Slack destination plugin in Chronosphere Telemetry Pipeline to send logs to Slack. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Slack [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `slack`) lets you configure your telemetry pipeline to send specific types of data or events to your Slack channels, such as error alerts, system status updates, or performance metrics. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------- | --------- | -------------------------------------------------- | ------------------------------------------------------------------------------- | | **Slack Webhook Address** | `webhook` | Absolute address of the webhook provided by Slack. | `https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX` | # Splunk destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/splunk Configure the Splunk destination plugin in Chronosphere Telemetry Pipeline to send logs and metrics to Splunk. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Splunk [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `splunk`, alias: `splunk_obs` or `splunk_siem`) lets you configure your telemetry pipeline to send your telemetry data to Splunk. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | --------------------- | -------------- | ------------------------------------------------------------------------------------ | ----------- | | **Host** | `host` | Required. IP address or hostname of the target Splunk service. | `127.0.0.1` | | **Port** | `port` | Required. TCP port of the target Splunk service. | `8088` | | **Compress** | `compress` | Sets the payload compression mechanism. Accepted values: `gzip`, *none*. | *none* | | **Splunk HTTP Token** | `splunk_token` | Required. Specifies the authentication token for the HTTP Event Collector interface. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced | Name | Key | Description | Default | | -------------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Splunk Channel** | `channel` | The `X-Splunk-Request-Channel` header to send to the HTTP Event Collector. | *none* | | **Enable Splunk Send Raw** | `splunk_send_raw` | When enabled, the recorded keys and values are set in the top level of the map instead of under the event key. For more information, see [Raw events](#raw-events). Accepted values: `true`, `false`. | `false` | | **Event Key** | `event_key` | Specifies the key name that will be used to send a single value as part of the record. | *none* | | **Event Host** | `event_host` | Sets the host value to the event data. The value allows a record accessor pattern. | *none* | | **Event Source** | `event_source` | Sets the source value to assign to the event data. | *none* | | **Event Source Type** | `event_sourcetype` | Sets the `sourcetype` value to assign to the event data. | *none* | | **Event Source Type Key** | `event_sourcetype_key` | Sets a record key that will populate `sourcetype`. If the key is found, it will have precedence over the value set in `event_sourcetype`. | *none* | | **Event Index** | `event_index` | The name of the index by which the event data is to be indexed. | *none* | | **Event Index Key** | `event_index_key` | Sets a record key that will populate the `index` field. If the key is found, it will have precedence over the value set in `event_index`. | *none* | | **Event Field(s)** | `event_field` | Sets event fields for the record. This option can be set multiple times and the format is `key_name record_accessor_pattern`. | *none* | | **Proxy** | `proxy` | Specifies an HTTP Proxy. The expected format of this value is `http://host:port`. HTTPS isn't supported. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | ### Debugging | Name | Key | Description | Default | | --------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **HTTP Buffer Size** | `http_buffer_size` | Specifies the buffer size used to read the response from the Splunk HTTP service. This option is used for debugging purposes when it's required to read full responses. Response size grows depending of the number of records inserted. To set an unlimited amount of memory, set this value to `false`. Otherwise the value must be according to the Unit Size specification. | *none* | | **Enable HTTP Debug Bad Request** | `http_debug_bad_request` | If the server returns an HTTP `400 Bad Request` status code and this flag is enabled, it will print the full HTTP request and response to the stdout interface. Used for debugging purposes. Accepted values: `true`, `false`. | `false` | ## Raw events By default, the Splunk destination plugin sends data to the `/services/collector/event` Splunk endpoint. However, if **Enable Splunk Send Raw** is enabled, this plugin sends data to the `/services/collector/raw` endpoint instead. The `/services/collector/raw` endpoint doesn't support nested fields in events, and also ignores any settings defined in the **Event Field(s)** plugin parameter. Because of these limitations, Chronosphere only recommends sending raw events in specific scenarios, like for machine data or IoT device data. Additionally, you should only send raw events after you configure Splunk to receive them accordingly. For more information about sending raw events, see the Splunk [Format events for HTTP Event Collector](https://help.splunk.com/en/splunk-enterprise/get-started/get-data-in/10.0/get-data-with-http-event-collector/format-events-for-http-event-collector#Event_metadata) documentation. # Standard output destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/standard-output Configure the Standard output destination plugin in Chronosphere Telemetry Pipeline to send logs, metrics, and traces to standard output. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Standard output [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `stdout`) lets you output your telemetry data to the standard output stream of a command line interface. If this plugin is active, you can use the [**Pipeline Output**](/ingest/pipeline/navigate#pipeline-output-v2-pipelines) pane in the Chronosphere Telemetry Pipeline web interface to view the data being routed through your pipeline. When you use **Pipeline Output** while this plugin is active, your telemetry data is temporarily cached in the Telemetry Pipeline backend. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :--------------------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Advanced | Name | Key | Description | Default | | ---------- | -------- | ------------------------------------------------------------------------------------------------------------------------------ | ------- | | **Format** | `format` | Required. Specifies the data format to be used in the HTTP request body. Accepted values: `json`, `json_lines`, `json_stream`. | `json` | # Sumo Logic destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/sumologic Configure the Sumo Logic destination plugin in Chronosphere Telemetry Pipeline to send logs to Sumo Logic. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Sumo Logic [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `Sumologic`) lets you configure your telemetry pipeline to send your data to Sumo Logic through HTTP, allowing you to control the format and structure of your data. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------------- | ------ | ---------------------------------------------------------------------------- | -------------------------------- | | **Collector URL** | `host` | Required. Sumo Logic collector URL. | `collectors.au.sumologic.com` | | **URI** | `uri` | Required. Replace `[PrivateKey]` with the key for your Sumo Logic collector. | `/receiver/v1/http/[PrivateKey]` | ### Advanced | Name | Key | Description | Default | | -------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Headers** | `header` | Adds an HTTP header key-value pair. You can set multiple headers. | *none* | | **Compress** | `compress` | Sets the payload compression mechanism. Accepted values: `gzip`, *none*. | *none* | | **HTTP Proxy** | `proxy` | Specifies an HTTP Proxy. The expected format of this value is `http://host:port`. | *none* | | **Format** | `format` | Required. Specifies the data format to be used in the HTTP request body. Accepted values: `json`, `json_lines`, `json_stream`. | `json_lines` | | **JSON Date Format** | `json_date_format` | Specifies the format of the date. Accepted values: `double`, `epoch`, `iso8601` (for example, `2018-05-30T09:39:52.000681Z`), `java_sql_timestamp` (for example, `2018-05-30 09:39:52.000681`, which can be used with AWS Athena). | `iso8601` | | **JSON Date Key** | `json_date_key` | Specifies the name of the date field in output. | `timestamp` | | **Body Key** | `body_key` | Specifies the key that contains the body. | *none* | | **Header Tag** | `header_tag` | Sets an HTTP header whose value is the tag of the record. | *none* | | **Header Key** | `headers_key` | Specifies the key that contains the headers. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | # Syslog destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/syslog Configure the Syslog destination plugin in Chronosphere Telemetry Pipeline to send logs over Syslog. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Syslog [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `syslog`) lets you configure your telemetry pipeline to transmit your data to remote systems or services over the network. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ---------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ----------- | | **Host** | `host` | Required. Domain or IP address of the remote Syslog server. | `127.0.0.1` | | **Port** | `port` | Required. TCP or UDP port of the remote Syslog server. | `514` | | **Mode** | `mode` | Required. Selected transport type. Accepted values: `tcp`, `tls`, `udp`. | `udp` | | **Syslog RFC Format** | `syslog_format` | Required. The Syslog protocol format to use. Accepted values: `rfc3164`, `rfc5424`. | `rfc5424` | | **Syslog Message Key** | `syslog_message_key` | Required. This is the key name from the original record that contains the message to deliver. Required, otherwise the message is empty. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ | | **Syslog Severity Key** | `syslog_severity_key` | Optional: The key name from the original record that contains the Syslog severity number. | *none* | | **Syslog Severity Preset** | `syslog_severity_preset` | Optional: The preset severity number. It will be overwritten if `syslog_severity_key` is set and a key of a record is matched. | *none* | | **Syslog Facility Key** | `syslog_facility_key` | Optional: The key name from the original record that contains the Syslog facility number. | *none* | | **Syslog Facility Preset** | `syslog_facility_preset` | Optional: The preset facility number. It will be overwritten if `syslog_facility_key` is set and a key of a record is matched. | *none* | | **Syslog Hostname Key** | `syslog_hostname_key` | Optional: The key name from the original record that contains the hostname that generated the message. | *none* | | **Syslog Hostname Preset** | `syslog_hostname_preset` | Optional: The preset hostname. It will be overwritten if `syslog_hostname_key` is set and a key of a record is matched. | *none* | | **Syslog Appname Key** | `syslog_appname_key` | Optional: The key name from the original record that contains the app name that generated the message. | *none* | | **Syslog Appname Preset** | `syslog_appname_preset` | Optional: The preset app name. It will be overwritten if `syslog_appnam_key` is set and a key of a record is matched. | *none* | | **Syslog ProcID Key** | `syslog_procid_key` | Optional: The key name from the original record that contains the Process ID that generated the message. | *none* | | **Syslog ProcID Preset** | `syslog_procid_preset` | Optional: The preset ProcID. It will be overwritten if `syslog_procid_key` is set and a key of a record is matched. | *none* | | **Syslog Message ID Key** | `syslog_msgid_key` | Optional: The key name from the original record that contains the Message ID associated to the message. | *none* | | **Syslog Message ID Preset** | `syslog_msgid_preset` | Optional: The preset message ID. It will be overwritten if `syslog_msgid_key` is set and a key of a record is matched. | *none* | | **Syslog Structured Data (SD) Key** | `syslog_sd_key` | Optional: The key name from the original record that contains the Structured Data (SD) content. | *none* | | **Syslog Maxsize** | `syslog_maxsize` | The maximum size allowed per message, as an integer in bytes. | Default size based on the `syslog_format` value: `rfc3164` sets this to `1024`, and `rfc5424` sets this to `2048`. | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # TCP destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/tcp Configure the TCP destination plugin in Chronosphere Telemetry Pipeline to send logs over TCP. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The TCP [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `tcp`) lets you configure your telemetry pipeline to send your data to a remote TCP endpoint, letting you integrate your telemetry data with other systems or services for various purposes. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ---------- | -------- | ------------------------------------------------------------------------------------------- | ----------- | | **Host** | `host` | Required. IP address or hostname of the target service. | `127.0.0.1` | | **Port** | `port` | Required. TCP port of the target service. | `80` | | **Format** | `format` | Required. Sets the format. Accepted values: `json`, `json_lines`, `json_stream`, `mspgack`. | `msgpack` | ### Advanced | Name | Key | Description | Default | | -------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | | **JSON Date Format** | `json_date_format` | Specifies the format of the date. Accepted values: `double`, `epoch`, `iso8601` (for example, 2018-05-30T09:39:52.000681Z) `java_sql_timestamp` (for example, 2018-05-30 09:39:52.000681). | *none* | | **JSON Date Key** | `json_date_key` | Specifies the name of the time key in the output record. To disable the time key, set the value to false. | *none* | | **Raw Message Key** | `raw_message_key` | Specifies the key that contains the message to send in raw format. For example, log key. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # UDP destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/udp Configure the UDP destination plugin in Chronosphere Telemetry Pipeline to send logs over UDP. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The UDP [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `udp`) provides a lightweight and efficient mechanism for transmitting your data, making it ideal for scenarios where low latency and high throughput are critical. Use this plugin to configure your pipeline to send data to a remote UDP endpoint, which lets you integrate your telemetry data with other systems or services. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ---------- | -------- | ------------------------------------------------------------------------------------------- | ----------- | | **Host** | `host` | Required. IP address or hostname of the target service. | `127.0.0.1` | | **Port** | `port` | Required. TCP port of the target service. | `80` | | **Format** | `format` | Required. Sets the format. Accepted values: `json_lines`, `json`, `json_stream`, `msgpack`. | `msgpack` | ### Advanced | Name | Key | Description | Default | | -------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **JSON Date Format** | `json_date_format` | Specifies the format of the date. Accepted values: `double`, `epoch`, `iso8601` (for example, `2018-05-30T09:39:52.000681Z`), `java_sql_timestamp` (for example, `2018-05-30 09:39:52.000681`). | *none* | | **JSON Date Key** | `json_date_key` | Specifies the name of the time key in the output record. To disable the time key, set the value to `false`. | *none* | | **Raw Message Key** | `raw_message_key` | Specifies the key that contains the message to send in raw format. For example, `log key`. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | # VictoriaMetrics destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/victoriametrics Configure the VictoriaMetrics destination plugin in Chronosphere Telemetry Pipeline to send metrics to VictoriaMetrics. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The VictoriaMetrics [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `prometheus_remote_write`, alias: `VictoriaMetrics`) lets you configure your telemetry pipeline to send data to VictoriaMetrics. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :-----------------: | :--------------------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------- | ------ | ------------------------------------------------------------------- | ----------- | | **Host** | `host` | Required. IP address or hostname of the target HTTP Server. | `127.0.0.1` | | **Port** | `port` | Required. TCP port of the target HTTP Server. | `80` | | **URI** | `uri` | Required. Specifies an optional HTTP URI for the target web server. | `/` | ### Advanced | Name | Key | Description | Default | | --------------------------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------- | ------- | | **Headers** | `header` | Adds an HTTP header key-value pair. Multiple headers can be set. | *none* | | **Additional Prometheus Metric Labels** | `add_label` | Adds custom labels to all metrics exposed through the Prometheus exporter. You can have multiple of these fields. | *none* | | **HTTP Proxy** | `proxy` | Specifies an HTTP Proxy. The expected format of this value is `http://host:port`. | *none* | | **Log Response Payload** | `log_response_payload` | Logs the response payload within the Core Pipeline log. | `false` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced Networking | Name | Key | Description | Default | | --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* | | **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* | | **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` | | **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` | | **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` | | **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` | | **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` | | **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* | | **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` | ### Basic Authentication | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Basic auth username. | *none* | | **HTTP Password** | `http_passwd` | Basic auth password. Requires `http_user` to be set. | *none* | # Vivo destination plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/destination-plugins/vivo Configure the Vivo destination plugin in Chronosphere Telemetry Pipeline to send logs, metrics, and traces to Vivo. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Vivo [destination plugin](/ingest/pipeline/plugins/destination-plugins) (name: `http`, alias: `vivo`) lets you configure your telemetry pipeline to send your telemetry data to a remote HTTP endpoint exposed by Vivo Exporter. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :--------------------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Advanced | Name | Key | Description | Default | | ------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------ | --------------- | | **Vivo Host** | `host` | Required. IP address or hostname of the target HTTP Server. | `calyptia-vivo` | | **Port** | `port` | Required. TCP port of the target HTTP Server. | `5489` | | **URI** | `uri` | Required. Specifies an optional HTTP URI for the target web server. For example, `/something`. | `/sink` | | **Format** | `format` | Required. Specifies the data format to be used in the HTTP request body. Accepted values: `json`, `json_lines`, `json_stream`. | `json` | # Ingest telemetry data with source plugins Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins Configure Chronosphere Telemetry Pipeline to ingest input data from various sources. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. ```mermaid actions={false} theme={null} flowchart LR accTitle: Pipeline flow accDescr: A simplified representation of a telemetry pipeline, which includes a source plugin, parser, processing rules, and destination plugin. subgraph Pipeline flow direction LR A(Source plugin) B(Parser) C(Processing rules) D(Destination plugin) A --> B --> C --> D style A stroke-width:4px end ``` Chronosphere Telemetry Pipeline uses *source plugins* to ingest telemetry data into pipelines, either by receiving it (push) or collecting it (pull). You can add and configure source plugins to connect your pipelines to a variety of telemetry sources. Different source plugins support different data types, including logs, metrics, and traces. ## Push-based and pull-based source plugins Source plugins are either push-based or pull-based. * **Push-based** source plugins passively listen for data emitted from a specified source. * **Pull-based** source plugins actively fetch data from a specified source. However, source plugins that output test data, like [Datagen](/ingest/pipeline/plugins/source-plugins/datagen), are classified as pull-based plugins even though they don't communicate with external sources. # Absolute source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/absolute Configure the Absolute source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Absolute [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http_loader`, alias: `absolute`) lets you retrieve data from Absolute and ingest it into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support the use of a [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the Pipeline Builder interface. This plugin doesn't support duplicates of itself within the same pipeline. Additionally, this plugin can't be used in combination with these source plugins within the same telemetry pipeline: * [HTTP API collector](/ingest/pipeline/plugins/source-plugins/http-collector) * [Mandiant ASM](/ingest/pipeline/plugins/source-plugins/mandiant-asm) * [Okta](/ingest/pipeline/plugins/source-plugins/okta) * [Signal Sciences](/ingest/pipeline/plugins/source-plugins/signal-sciences) * [Slack](/ingest/pipeline/plugins/source-plugins/slack) * [Vectra M365](/ingest/pipeline/plugins/source-plugins/vectram365) ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------ | ------ | ------------------------------------------------------ | --------------------------------------- | | **API Access URL** | `url` | Required. The URL used to access the Absolute API. | `https://api.absolute.com/jws/validate` | | **Token Key ID** | *none* | Required. The token access key for accessing Absolute. | *none* | | **Signing Key** | *none* | Required. The signing key for Absolute access. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # S3 Input (One Time) source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/amazon-s3 Configure the S3 Input (One Time) source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. > Requires [pipeline agent](/ingest/pipeline/v2/component-versions) v25.8.1 or later, > [Core Operator](/ingest/pipeline/v2/component-versions) v3.67.0 or later, and > [Pipeline CLI](/ingest/pipeline/pipeline-cli) v3.66.0 or later. The S3 Input (One Time) [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `go-s3-replay-plugin`) lets you ingest data from an Amazon S3 bucket into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support duplicates of itself within the same pipeline. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Requirements To use the S3 Input (One Time) plugin, you must meet these requirements: * Your IAM user or IAM role must have the [`s3:GetObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) permission for your S3 bucket. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ------------------------ | ---------------------- | -------------------------------------------------------------------------------------------------- | ------- | | **AWS Bucket Name** | `aws_bucket_name` | Required. The name of the S3 bucket where your data is stored. | *none* | | **AWS Bucket Region** | `aws_bucket_region` | Required. The name of the AWS region where your S3 bucket exists. For example: `us-east1`. | *none* | | **File Path** | `logs` | Required. The glob format of the logs stored in your S3 bucket. For example: `/logs/\*.log`. | *none* | | **Line Buffer Max Size** | `max_line_buffer_size` | The maximum line size the plugin will read from [JSON or plain text files](#supported-data-types). | `10MiB` | ### Advanced | Name | Key | Description | Default | | :---------------------- | :-------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------ | | **S3 Assume Role ARN** | `assume_role_arn` | The ARN of the role to assume in S3. This can be an ARN within the same account or across accounts. | *none* | | **S3 Role External ID** | `role_external_id` | The [external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html) of the role to assume in S3. | *none* | | **S3 Endpoint** | `aws_s3_endpoint` | The URL of your S3 endpoint. | *none* | | **S3 Read Concurrency** | `s3_read_concurrency` | The maximum number of concurrent S3 [`GetObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) calls that this plugin will make. | The number of logical CPUs allocated to each pipeline [replica](/ingest/pipeline/v2/configure/scaling). | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ## Authentication methods The S3 Input (One Time) plugin supports the following authentication methods: * [EKS Pod Identities](#eks-pod-identities) * [IMDS](#imds) * [IRSA](#irsa) * [Static credentials](#static-credentials) ### EKS Pod Identities To use EKS Pod Identities for authentication: 1. In AWS, configure [EKS Pod Identities](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html). 2. In [Pipeline CLI](/ingest/pipeline/pipeline-cli), add the following flag to a `create pipeline` or `update pipeline` command: ```shell /VALUE/ theme={null} calyptia {create|update} pipeline --service-account VALUE ``` Replace *`VALUE`* with the name of the Kubernetes service account associated with your Pods. ### IMDS To use IMDS for authentication: * In AWS, configure [IAM roles for your EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html). ### IRSA To use IRSA for authentication: 1. In AWS, [set up IRSA for your EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html). 2. [Assign an IAM role to your Kubernetes service account](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html). 3. In [Pipeline CLI](/ingest/pipeline/pipeline-cli), add the following flag to a `create pipeline` or `update pipeline` command: ```shell /VALUE/ theme={null} calyptia {create|update} pipeline --service-account VALUE ``` Replace *`VALUE`* with the name of your Kubernetes service account. ### Static credentials To use static credentials for authentication: * In Telemetry Pipeline, create [create secrets](/ingest/pipeline/v2/configure/secrets#add-a-secret) that contain the values of your [IAM access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). These secrets must use the key names `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. You don't need to add an explicit [reference](/ingest/pipeline/v2/configure/secrets#reference-a-secret) to these secrets in your pipeline configuration file. If secrets with the correct key names are present, the S3 Input (One Time) plugin automatically detects these values and uses them for authentication. ## Supported data types The S3 Input (One Time) plugin can ingest [JSON objects](#json) and [plain text](#plain-text) from files stored in S3 buckets, including gzip-compressed files. Additionally, this plugin can extract and ingest compressed and uncompressed files from [tar archives](#tar-archives). ### JSON This plugin can ingest data from JSON files with these file extensions: * `.json` * `.jsonl` * `.ndjson` If a file contains only a single JSON object, this plugin creates a new log from that object. If a file contains multiple [newline-delimited JSON (NDJSON)](https://github.com/ndjson/ndjson-spec) objects, this plugin creates a new log from each JSON object within that file. Key-value pairs from JSON objects are stored as key-value pairs in the resulting log. For JSON files that use gzip compression (with file extensions such as `.json.gzip` or `.json.gz`), this plugin decompresses each file before processing it accordingly. ### Plain text If a file doesn't use a file extension that identifies it as a JSON file, the S3 Input (One Time) plugin processes that file as plain text. It creates a new log from each line of the file and stores the content in a key named `_raw` within the resulting log. For non-JSON files that use gzip compression (with file extensions that include the `.gzip` or `.gz` suffix), this plugin decompresses each file before processing it accordingly. ### Tar archives The plugin can extract and consume files from tar archives with these file extensions: * `.tar` * `.tar.gz` * `.tar.gzip` After the plugin extracts these files, it processes any [JSON](#json) and [plain text](#plain-text) data accordingly, but skips directories and symbolic links. If files inside a tar archive are gzip-compressed, this plugin decompresses those files accordingly. ## Metadata The S3 Input (One Time) plugin attaches the following metadata to each log: * `__chrono_bucket`: The name of the S3 bucket that contains the file from which the log was created. * `__chrono_file`: The key of the S3 object from which the log was created. * `__chrono_tar_file_entry`: For data extracted from [tar archives](#tar-archives) only. The name of the tar archive that contained the file from which the log was created. ## Get started To get started with the S3 Input (One Time) plugin, follow these steps. 1. Either [create a new pipeline](/ingest/pipeline/v2/build/create-modify#create-a-pipeline) or [modify an existing pipeline](/ingest/pipeline/v2/build/create-modify). 2. For testing purposes, set the pipeline's destination to [standard output](/ingest/pipeline/plugins/destination-plugins/standard-output). 3. Set the pipeline's source to S3 Input (One Time), and then add values for all required parameters, along with any optional parameters of your choosing. 4. Set up one of the supported [authentication methods](#authentication-methods) for the S3 Input (One Time) source plugin. 5. In the Telemetry Pipeline web interface, go to the summary page for that pipeline. 6. In the [**Pipeline Output**](/ingest/pipeline/navigate#pipeline-output-v2-pipelines) section, click **Get latest logs**. 7. Review this log output to ensure that you're receiving data from S3. If you don't receive any data, or if you encounter connection errors, review your plugin configuration settings. 8. After you've confirmed that the S3 Input (One Time) plugin is functioning correctly, you can overwrite the standard output destination with the destination where you want to send your telemetry data. # Amazon Kinesis Firehose source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/aws-firehose Receive pipeline data from Amazon Kinesis Firehose. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Amazon Kinesis Firehose [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http`, alias: `AWS_Kinesis_Firehose`) lets you retrieve data from your Amazon Kinesis Firehose instances and ingest it into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------- | ------ | ----------------------------------------------------------------------- | ------- | | **Port** | `port` | Required. The port to listen to for incoming Kinesis Firehose messages. | `9880` | ### Advanced | Name | Key | Description | Default | | ---------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Tag Key** | `tag_key` | Specifies the key name to overwrite a tag. If set, the value of that key overwrites the tag. | *none* | | **Buffer Max Size** | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin drops that record. | `4M` | | **Buffer Chunk Size** | `buffer_chunk_size` | Sets the default [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin temporarily increases the chunk size up to the value of `buffer_max_size` to accommodate it. | `512k` | | **Successful Response Code** | `successful_response_code` | Sets a successful response code. Accepted values: `200`, `201`, `204`. | `200` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | # Amazon Kinesis Stream Input source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/aws-streams Receive pipeline data from Amazon Kinesis Stream Input. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Amazon Kinesis Stream Input [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `aws_kinesis_stream`) lets you retrieve data from Amazon Kinesis Stream and ingest it into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support duplicates of itself within the same pipeline. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | --------------------------- | ----------------------- | --------------------------------------------------------------- | ------- | | **AWS Access Key** | `aws_access_key_id` | Required. Your AWS access key path. | *none* | | **AWS Secret Key** | `aws_secret_access_key` | Required. Your AWS secret key path. | *none* | | **AWS Kinesis Stream Name** | `stream_name` | Required. The AWS Kinesis stream name. | *none* | | **AWS Region** | `aws_region` | Required. The name of the AWS region. For example, `us-east-1`. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Number of fetched records** | `limit` | The maximum number of records to fetch from the stream on each iteration. | `1000` | | **Pull Interval** | `empty_interval` | How long to wait before fetching more records. If the stream returns empty, Telemetry Pipeline will continue to pull. | `1s` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # Azure Event Hubs source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/azure-eventhub Configure the Azure Event Hubs source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Azure Event Hubs [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `kafka`, alias: `Azure_Event_Hub`) lets you ingest data from your Azure Event Hubs instances into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------- | | **Event Hub Namespace** | `brokers` | Required. Your Event Hub namespace. | `[REPLACE WITH YOUR NAMESPACE].servicebus.windows.net:9093` | | **Event Hub Name** | `topics` | Required. The name of the event hub (equivalent to a Kafka topic) to read information from. | *none* | | **Connection String Key** | `rdkafka.sasl.password` | Required. The Event Hub connection string from within the connection access policy set for the source. | `Endpoint=[REPLACE WITH YOUR CONNECTION STRING VALUE]` | ### Advanced | Name | Key | Description | Default | | --------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | | **Minimum Queued Messages** | `rdkafka.queued.min.messages` | Minimum number of messages per event hub and partition that Telemetry Pipeline tries to maintain in the local consumer queue. | `10` | | **Request Timeout (ms)** | `rdkafka.request.timeout.ms` | How long Telemetry Pipeline waits before terminating a request connection. Recommended value: `60000`. | `60000` | | **Session Timeout (ms)** | `rdkafka.session.timeout.ms` | How long Telemetry Pipeline waits before prior to terminating a session connection. Recommended value: `30000`. | `30000` | | **SASL Username** | `rdkafka.sasl.username` | SASL username. | `$ConnectionString` | | **Security Protocol** | `rdkafka.security.protocol` | The security protocol for Azure Event Hub. If you require OAuth 2.0 or OpenID authentication, contact Chronosphere Support. | `SASL_SSL` | | **SASL Mechanism** | `rdkafka.sasl.mechanism` | The transport mechanism for the SASL connection. | `PLAIN` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Other This parameter doesn't have an equivalent setting in the Telemetry Pipeline web interface, but you can use it in pipeline configuration files. | Name | Key | Description | Default | | ------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | *none* | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single log exceeds this size, the plugin drops that log. | `4M` | ### Extended librdkafka parameters This plugin uses the [librdkafka](https://github.com/confluentinc/librdkafka) library. Certain configuration parameters available through the Telemetry Pipeline UI are based on librdkafka settings. These parameters generally use the `rdkafka.` prefix. In addition to the parameters available through the Telemetry Pipeline UI, you can customize any of the [librdkafka configuration properties](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) by adding them to a pipeline configuration file. To do so, append the `rdkafka.` prefix to the name of that property. For example, to customize the `socket.keepalive.enable` property, add the `rdkafka.socket.keepalive.enable` key to your configuration file. Don't use librdkafka properties to configure a pipeline's memory buffer. Instead, use the [`buffer_max_size`](#other) parameter. # Azure Event Grid source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/azure-grid Configure the Azure Event Grid source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Azure Event Grid [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `azeventgrid`) lets you receive data from Azure and ingest it into a telemetry pipeline using the [pull delivery](https://learn.microsoft.com/azure/event-grid/pull-delivery-overview) API. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support the use of a [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the Pipeline Builder interface. This plugin doesn't support duplicates of itself within the same pipeline. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | --------------------------- | ----------------------- | ------------------------------------------------------------------- | ------- | | **Endpoint** | `endpoint` | Required. The endpoint URL of your Event Grid instance's namespace. | *none* | | **Shared Key** | `key` | Required. The shared key used for authentication. | *none* | | **Topic Name** | `topicName` | Required. The topic name from which to receive events. | *none* | | **Event Subscription Name** | `eventSubscriptionName` | Required. The event subscription name. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ## Example configuration ```yaml theme={null} service: log_level: error pipeline: inputs: - name: azeventgrid endpoint: https://myNamespaceName.westus-1.eventgrid.azure.net key: ${SECRET_AZEVENTGRID_KEY} topicName: myTopic eventSubscriptionName: myEventSubscription outputs: - name: stdout match: "*" ``` # Bash Command source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/bash Run a bash command in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Bash Command [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `exec`) lets you run bash commands once at pipeline startup, or at repeated intervals. Although this is a self-contained plugin that doesn't communicate with external sources, it's still classified as a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ------------------------------ | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Command** | `command` | Required. The bash command to run. | *none* | | **Polling Interval (Seconds)** | `interval_sec` | Required. The number of seconds between polls. | *none* | | **Only Run Once** | `Oneshot` | Allows collection of data preceding pipelines startup. If `true`, this plugin runs only a single time at startup. Accepted values: `true`, `false`. | `false` | ### Advanced | Name | Key | Description | Default | | ---------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Buffer Size** | `Buf_Size` | The buffer sizes for your bash command. In cases of large output, set this value to a larger size. | `8mb` | | **Polling Interval (nanoseconds)** | `Interval_Nsec` | The number of nanoseconds to add to the polling interval set in the [**Required**](#required) settings. | `0` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # Cloudflare LogPush source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/cloudflare Configure the Cloudflare LogPush source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. > Requires [pipeline agent](/ingest/pipeline/v2/component-versions) v25.3.8 or later. The Cloudflare LogPush [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `cloudflare`) lets you ingest data from Cloudflare LogPush jobs into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support duplicates of itself within the same pipeline. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | **Port** | `addr` | The port where Telemetry Pipeline will listen for incoming data. In the Telemetry Pipeline web interface, you only need to specify the actual port number. In pipeline configuration files, this value combines the network interface and port number. The network interface for this plugin is always `0.0.0.0`. | `9880` in the web interface, `0.0.0.0:9880` in pipeline configuration files | ### Security and TLS | Name | Key | Description | Default | | ------------------- | ------------- | ----------------------------------------------------------- | ------- | | **HTTP Username** | `http_user` | Your username for HTTP basic authentication, if applicable. | *none* | | **HTTP Password** | `http_passwd` | Your password for HTTP basic authentication, if applicable. | *none* | | **TLS Certificate** | `cert_file` | The path to your TLS certificate file, if applicable. | *none* | | **TLS Key** | `key_file` | The path to your TLS key file, if applicable. | *none* | ### Ownership Challenge | Name | Key | Description | Default | | ----------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | | **Skip Ownership Challenge** | `skipownershipchallenge` | If `false`, this plugin will trigger a [Cloudflare ownership challenge](#ownership-challenge). If `true`, this plugin won't trigger an ownership challenge. | `true` | | **Cloudflare API Key or API Token** | `cloudflareapikey` | Required if **Skip Ownership Challenge** is `false`. The Cloudflare API key or API token to use in the ownership challenge. This parameter supports plain text values and the environment variables `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_API_KEY`. | *none* | | **Cloudflare Email** | `cloudflareemail` | Required if **Skip Ownership Challenge** is `false` and if using key-based authentication, but not token-based authentication. The email address associated with your Cloudflare account. In addition to plain text values, this parameter supports the environment variable `CLOUDFLARE_EMAIL`. | *none* | | **Destination** | `destination` | Required if **Skip Ownership Challenge** is `false`. The HTTP destination that you configured for your LogPush job. This parameter supports plain text values and the environment variable `CLOUDFLARE_DESTINATION`. | *none* | | **Cloudflare Account ID** | `cloudflareaccountid` | Required if **Skip Ownership Challenge** is `false` and if **Cloudflare Zone ID** is unset. The Cloudflare account ID associated with your LogPush job. This parameter supports plain text values and the environment variable `CLOUDFLARE_ACCOUNT_ID`. | *none* | | **Cloudflare Zone ID** | `cloudflarezoneid` | Required if **Skip Ownership Challenge** is `false` and if **Cloudflare Account ID** is unset. The Cloudflare zone ID associated with your LogPush job. This parameter supports plain text values and the environment variable `CLOUDFLARE_ZONE_ID`. | *none* | ### Advanced | Name | Key | Description | Default | | ---------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | | **Response Headers** | `resp_headers` | The header for Telemetry Pipeline to include in responses to Cloudflare. To include multiple headers, include each header as a new line. Supports templating. | `Content-Type:application/json` | | **Response Status Code** | `resp_status_code` | The HTTP status code for Telemetry Pipeline to include in responses to Cloudflare. | `200` | | **Response Body** | `resp_body` | The response body for Telemetry Pipeline to send to Cloudflare. Supports templating. | `{"status": "ok"}` | | **Time Extraction Template** | `time_from` | The template to extract time data from logs. For example, if your logs use the RFC3339 time format, set the value `{{.Format time.RFC3339Nano}}`. If no value is set, Telemetry Pipeline assigns a timestamp to each log based on when Telemetry Pipeline *receives* that log. | *none* | | **Cloudflare Base URL** | `baseurl` | Required if **Skip Ownership Challenge** is `false`. The Cloudflare API base URL to use in the ownership challenge. This parameter supports plain text values and the environment variable `CLOUDFLARE_BASE_URL`. | `https://api.cloudflare.com/client/v4/` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ## LogPush jobs For information about setting up Cloudflare LogPush jobs, consult the Cloudflare LogPush [Enable HTTP destination](https://developers.cloudflare.com/logs/get-started/enable-destinations/http/) guide. When you create a new LogPush job in Cloudflare, there might be a delay before data from that job starts flowing to Telemetry Pipeline. Similarly, when you delete a LogPush job in Cloudflare, there might be a delay before data stops flowing. These delays can range anywhere from a few minutes to 15 minutes. ## Ownership challenge You can set **Skip Ownership Challenge** to `false` to trigger a one-time [Cloudflare ownership challenge](https://developers.cloudflare.com/api/resources/logpush/subresources/ownership/methods/create/) for enhanced security and safety. However, this ownership challenge is optional, and it's possible to send LogPush data to Telemetry Pipeline without it. If you trigger an ownership challenge outside of Telemetry Pipeline, the ensuing challenge response will appear in your pipeline logs, including tokens, and will be sent to any of your pipeline's active destinations. This behavior occurs even when the **Skip Ownership Challenge** setting in Telemetry Pipeline is set to `true`. # Confluent Cloud source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/confluent-cloud Configure the Confluent Cloud source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Confluent Cloud [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `kafka`, alias: `confluent_cloud`) lets you retrieve data from Confluent Cloud and ingest it into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------------------- | ----------------------- | -------------------------------------------------------------------------------- | ---------------------------------------------- | | **Confluent Cloud Bootstrap Servers** | `brokers` | Required. The Confluent Cloud bootstrap found within the configuration settings. | `[YOUR_BOOTSTRAP_SERVER].confluent.cloud:9092` | | **Confluent Cloud Topic** | `topics` | Required. The Confluent Cloud topic to read information from. | *none* | | **Confluent Cloud API Key** | `rdkafka.sasl.username` | Your Confluent Cloud API key. | *none* | | **Confluent Cloud API Secret** | `rdkafka.sasl.password` | Required. Your Confluent Cloud API secret. | *none* | ### Advanced | Name | Key | Description | Default | | --------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | **Minimum Queued Messages** | `rdkafka.queued.min.messages` | The minimum number of messages per topic and partition that Telemetry Pipeline tries to maintain in the local consumer queue. | `10` | | **Session Timeout (ms)** | `rdkafka.session.timeout.ms` | How long Telemetry Pipeline waits before terminating a session connection. | `45000` | | **Security Protocol** | `rdkafka.security.protocol` | The security protocol for Azure Event Hub. If you require OAuth 2.0 or OpenID authentication, contact Chronosphere Support. | `SASL_SSL` | | **SASL Mechanism** | `rdkafka.sasl.mechanism` | The transport mechanism for the SASL connection. | `PLAIN` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Other This parameter doesn't have an equivalent setting in the Telemetry Pipeline web interface, but you can use it in pipeline configuration files. | Name | Key | Description | Default | | ------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | *none* | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single log exceeds this size, the plugin drops that log. | `4M` | ### Extended librdkafka parameters This plugin uses the [librdkafka](https://github.com/confluentinc/librdkafka) library. Certain configuration parameters available through the Telemetry Pipeline UI are based on librdkafka settings. These parameters generally use the `rdkafka.` prefix. In addition to the parameters available through the Telemetry Pipeline UI, you can customize any of the [librdkafka configuration properties](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) by adding them to a pipeline configuration file. To do so, append the `rdkafka.` prefix to the name of that property. For example, to customize the `socket.keepalive.enable` property, add the `rdkafka.socket.keepalive.enable` key to your configuration file. Don't use librdkafka properties to configure a pipeline's memory buffer. Instead, use the [`buffer_max_size`](#other) parameter. # Datadog Agent source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/datadog-agent Use the Datadog Agent source plugin to ingest log data. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Datadog Agent [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http`, alias: `datadog_agent`) ingests data from Datadog into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------- | ------ | ------------------------------------------------------- | ------- | | **Port** | `port` | Required. The port for Telemetry Pipeline to listen on. | `9880` | ### Advanced | Name | Key | Description | Default | | ---------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Buffer Max Size** | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin drops that record. | `4M` | | **Buffer Chunk Size** | `buffer_chunk_size` | Sets the default [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin temporarily increases the chunk size up to the value of `buffer_max_size` to accommodate it. | `512K` | | **Successful Response Code** | `successful_response_code` | Sets the code returned by successful responses. Accepted values: `200`, `201`, `204`. | `200` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | # Datagen source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/datagen Use the Datagen source plugin to generate simulated log data to test your pipeline configuration. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Datagen [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `datagen`, alias: `apache_common`) generates simulated log data through [Gofakeit](https://github.com/brianvoe/gofakeit). You can use this simulated data to test your pipeline configurations. Although this is a self-contained plugin that doesn't communicate with external sources, it's still classified as a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support the use of a [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the Pipeline Builder interface. This plugin doesn't support duplicates of itself within the same pipeline. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------ | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Template** | `template` | Required. A JSON object that specifies how to generate log data. Accepted types: Apache common, Syslog rfc3164, or Syslog rfc5424. See [Templates](#templates). | *none* | | *none* | `rate` | How often to generate logs, in seconds. | `1` | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ## Templates Templates define the structure and content of your simulated log data. When you configure the Datagen source plugin in a pipeline's YAML configuration file, use the `template` key to create a JSON object. For example: ```yaml theme={null} pipeline: inputs: - name: datagen template: |- { "host": "{{ IPv4Address }}", "user": { "user-identifier": "{{ Username }}", "auth-user-id": "{{ UUID }}", } "method": "{{ HTTPMethod }}", "request": "{{ URLPath }}", "protocol": "{{ HTTPVersion }}", "response-code": "{{ HTTPStatusCode }}", "bytes": "{{ Number 0 30_000 }}" } ``` When the Datagen plugin generates logs, each log matches the format of your template, but any included [functions](#functions) are replaced by simulated data. For best results, Chronosphere recommends creating a JSON object that uses custom strings for its keys and functions for its values. ## Functions The Datagen plugin uses functions to generate randomized values that follow a consistent pattern. For example, `{{ HTTPMethod }}` can output a string like `GET`, `POST`, or `PUT`, but not a string like `GREEN` or `SUNDAY`. To invoke a function in a [template](#templates), surround the name of that function with double curly braces. ### Suggested functions You can use any of the available [Gofakeit functions](https://github.com/brianvoe/gofakeit#functions) in a template. The following functions are best suited for generating realistic log data: * `{{ HTTPMethod }}`: Returns an HTTP method, like `GET`. * `{{ HTTPStatusCode }}`: Returns an HTTP status code, like `403`. * `{{ HTTPVersion }}`: Returns an HTTP protocol version, like `HTTP/1.1`. * `{{ IPv4Address }}`: Returns a random IPv4 address. * `{{ IPv6Address }}`: Returns a random IPv6 address. * `{{ Username }}`: Returns a random username, like `Bailey1270`. * `{{ UUID }}`: Returns a string of numbers in UUID format. * `{{ Date }}`: Returns a random time in UTC format. * `{{ LogLevel syslog }}`: Returns a log type, like `alert`. * `{{ Number 0 30_000 }}`: Returns a number between 0 and 30,000. * `{{ URLPath }}`: Returns a string that resembles a URL path, like `/foo/bar`. This function is unique to Datagen and isn't included in Gofakeit. * `{{ URLScheme }}`: Returns either `http` or `https`. This function is unique to Datagen and isn't included in Gofakeit. ## Example output For a pipeline with the following template: ```yaml theme={null} pipeline: inputs: - name: datagen template: |- { "response-code": "{{ HTTPStatusCode }}", "bytes": "{{ Number 0 30_000 }}" "host": "{{ IPv4Address }}", } ``` The Datagen source plugin generates simulated log data in the following format: ```shell theme={null} {"response-code"=>201.000000, "bytes"=>29600.000000, "host"=>"64.15.54.123"} {"response-code"=>302.000000, "bytes"=>429.000000, "host"=>"214.221.240.201"} {"response-code"=>205.000000, "bytes"=>3507.000000, "host"=>"27.79.197.250"} {"response-code"=>504.000000, "bytes"=>758.000000, "host"=>"211.135.114.51"} {"response-code"=>404.000000, "bytes"=>12190.000000, "host"=>"84.74.177.0"} ``` # Dummy logs, metrics, and traces source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/dummy Generate sample telemetry data for Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Dummy logs, metrics, and traces [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `event_type`) lets you create placeholder telemetry data for Telemetry Pipeline. Although this is a self-contained plugin that doesn't communicate with external sources, it's still classified as a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :--------------------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ------------------ | ------ | ----------------------------------------------------------------------------------------------- | ------- | | **Telemetry Type** | `type` | Required. The type of telemetry data to generate. Accepted values: `logs`, `metrics`, `traces`. | `logs` | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # Dummy HTTP source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/dummy-http Generate sample telemetry data for Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Dummy HTTP [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `datagen`, alias: `dummy_http`) generates simulated log data that you can use to test your pipeline configuration. This plugin is based on the [Datagen source plugin](/ingest/pipeline/plugins/source-plugins/datagen) and uses a [predefined template](#template-format) to generate its logs. Although this is a self-contained plugin that doesn't communicate with external sources, it's still classified as a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support the use of a [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the Pipeline Builder interface. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ## Template format Logs generated by the Dummy HTTP source plugin use the following template format: ```yaml theme={null} template: >- { "log": "{{ IPv4Address }} - {{ FirstName }} [{{ Day }}/{{ MonthString }}/{{ Number 2011 2024 }}:{{ Hour }}:{{ Minute }}:{{ Second }} -0700] \"{{ HTTPMethod }} {{ URLPath }} HTTP/{{ HTTPVersion }}\" {{ HTTPStatusCode }} {{ Number 100 10000 }} \"{{ URL }}\" \"{{ UserAgent }}\"" } ``` This template produces logs that resemble the following examples: ```json theme={null} [{"date":1733159068.797057,"log":"142.201.230.54 - Kitty [31/February/1938:14:22:50 -0700] \"HEAD /user-centric/open-source HTTP/HTTP/1.1\" 301 7907 \"https://www.principalunleash.name/magnetic/end-to-end/eyeballs\" \"Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_8_5 rv:4.0; en-US) AppleWebKit/535.49.6 (KHTML, like Gecko) Version/6.2 Safari/535.49.6\""}] [{"date":1733159069.797646,"log":"165.214.42.13 - Riley [8/July/1998:16:52:34 -0700] \"GET /enterprise/intuitive HTTP/HTTP/1.1\" 406 6981 \"https://www.productdot-com.com/enable/efficient/content/brand\" \"Mozilla/5.0 (iPad; CPU OS 8_0_3 like Mac OS X; en-US) AppleWebKit/534.13.6 (KHTML, like Gecko) Version/4.0.5 Mobile/8B113 Safari/6534.13.6\""}] [{"date":1733159070.796631,"log":"13.114.140.228 - Bailey [1/February/1989:18:11:27 -0700] \"HEAD /distributed HTTP/HTTP/1.0\" 401 9871 \"https://www.internationalmorph.biz/24-365/global/action-items\" \"Mozilla/5.0 (iPad; CPU OS 7_3_3 like Mac OS X; en-US) AppleWebKit/532.14.2 (KHTML, like Gecko) Version/4.0.5 Mobile/8B119 Safari/6532.14.2\""}] [{"date":1733159071.799888,"log":"72.109.188.94 - Fabiola [14/May/1987:0:43:25 -0700] \"DELETE /e-enable/virtual HTTP/HTTP/2.0\" 100 3605 \"https://www.internationalinnovative.com/bandwidth\" \"Opera/8.34 (Macintosh; U; PPC Mac OS X 10_6_7; en-US) Presto/2.12.268 Version/11.00\""}] [{"date":1733159072.797658,"log":"19.32.161.243 - Marlene [23/March/1917:3:0:19 -0700] \"PATCH /global/deliverables HTTP/HTTP/1.1\" 205 6830 \"https://www.centralenable.info/e-business/vertical/models\" \"Mozilla/5.0 (Windows NT 5.0) AppleWebKit/5320 (KHTML, like Gecko) Chrome/38.0.815.0 Mobile Safari/5320\""}] [{"date":1733159073.797684,"log":"40.66.86.63 - Leif [12/March/1904:6:28:31 -0700] \"POST /dot-com HTTP/HTTP/1.0\" 304 3638 \"http://www.humansexy.org/transparent\" \"Mozilla/5.0 (Windows NT 5.01; en-US; rv:1.9.2.20) Gecko/1915-02-18 Firefox/35.0\""}] [{"date":1733159074.79653,"log":"196.27.18.191 - Shanny [10/January/1992:14:28:43 -0700] \"GET /systems/best-of-breed/incubate/enable HTTP/HTTP/2.0\" 403 1572 \"https://www.districtinitiatives.info/holistic/best-of-breed/integrated/24-7\" \"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/5352 (KHTML, like Gecko) Chrome/37.0.888.0 Mobile Safari/5352\""}] [{"date":1733159075.797568,"log":"154.13.34.48 - Neal [3/October/1931:22:16:42 -0700] \"GET /out-of-the-box/eyeballs/evolve/solutions HTTP/HTTP/1.1\" 200 7010 \"http://www.dynamicintegrate.io/incentivize/niches\" \"Mozilla/5.0 (Windows NT 6.0; en-US; rv:1.9.0.20) Gecko/1955-08-17 Firefox/36.0\""}] [{"date":1733159076.796681,"log":"220.237.28.126 - Matteo [2/October/1903:10:42:10 -0700] \"HEAD /integrate/efficient HTTP/HTTP/2.0\" 406 2280 \"http://www.productdisintermediate.name/mission-critical/visionary\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 8_1_2 like Mac OS X; en-US) AppleWebKit/536.51.1 (KHTML, like Gecko) Version/3.0.5 Mobile/8B111 Safari/6536.51.1\""}] [{"date":1733159077.800555,"log":"113.30.237.225 - Abigayle [16/October/1925:10:32:6 -0700] \"DELETE /world-class/expedite/architectures HTTP/HTTP/1.0\" 504 1560 \"https://www.principale-services.io/strategize/deliverables\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/5361 (KHTML, like Gecko) Chrome/40.0.884.0 Mobile Safari/5361\""}] ``` # Elasticsearch HTTP source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/elasticsearch-http Configure the Elasticsearch HTTP source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Elasticsearch HTTP [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `elasticsearch`) lets you ingest data from your Elasticsearch HTTP instances into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | -------- | ------ | ---------------------------------------------------------------- | ------- | | **Port** | `port` | Required. The TCP port used for listening for incoming messages. | `9200` | ### Advanced | Name | Key | Description | Default | | ----------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Buffer Chunk Size** | `buffer_chunk_size` | Sets the default [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin temporarily increases the chunk size up to the value of `buffer_max_size` to accommodate it. | `5M` | | **Buffer Max Size** | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin drops that record. | `20M` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | # Fluent Bit source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/fluent-bit Gather telemetry from Fluent Bit sources to send to Chronosphere Telemetry Pipeline for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Fluent Bit [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `forward`, alias: `Fluent_Bit`) lets you ingest data from your Fluent Bit instances into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :--------------------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------- | ------ | ------------------------------------------------------- | ------- | | **Port** | `port` | Required. The TCP port listening for incoming messages. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ## Configuring Fluent Bit to route data to Telemetry Pipeline You can use the `out_forward` Fluent Bit plugin to route data to the Fluent Bit data source in Chronosphere Telemetry Pipeline. In your Fluent Bit configuration, use the following configuration: ```yaml theme={null} [OUTPUT] Name forward Match * Host 127.0.0.1 Port 24244 ``` # Fluentd source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/fluentd Configure the Fluentd source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Fluentd [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `forward`, alias: `Fluentd`) lets you ingest data from your Fluentd instances into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------- | ------ | ---------------------------------------------------------------- | ------- | | **Port** | `port` | Required. The TCP port used for listening for incoming messages. | `5170` | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ## Configuring Fluentd to send data to Telemetry Pipeline You can configure Fluentd to send data to Chronosphere Telemetry Pipeline by using the `out_forward` Fluentd plugin. For more information, visit the [Fluentd documentation](https://docs.fluentd.org/output/forward). # Forward source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/forward Configure the Forward source plugin in Chronosphere Telemetry Pipeline to ingest logs, metrics, and traces for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Forward [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `forward`, alias: `forward`) lets you ingest data from your Forward services into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :--------------------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | **Host** | `listen` | Required. The listener network interface. | `0.0.0.0` | | **Port** | `port` | Required. The TCP port used for listening for incoming messages. | `24224` | | **Unix Socket Path** | `unix_path` | Specifies the path to Unix socket to receive a Forward message. If set, `listen` and `port` are ignored. | *none* | | **Unix Socket** | `unix_perm` | Sets the permission of the Unix socket file. If `unix_path` isn't set, this parameter is ignored. | *none* | | **Buffer Chunk Size** | `buffer_chunk_size` | Sets the default [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin temporarily increases the chunk size up to the value of `buffer_max_size` to accommodate it. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). | `1024000` | | **Buffer Max Size** | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin drops that record. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). | `6144000` | | **Tag Prefix** | `tag_prefix` | The prefix incoming tag with the defined value. | *none* | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | # GCP PubSub Input source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/gcp-pubsub Configure the GCP PubSub Input source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. > Requires Chronosphere Telemetry Pipeline v2.72.0 or later. The Google Cloud PubSub Input [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `gcp_pubsub`) ingests data from Google Cloud Pub/Sub into a telemetry pipeline. You can use this plugin to ingest [LogEntry](https://docs.cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) objects generated from a [GCP Logging Sink](https://docs.cloud.google.com/logging/docs/routing/overview#sinks) with a [pull-type](https://docs.cloud.google.com/pubsub/docs/subscriber) topic subscription. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Subscription ID** | `subscription_id` | Required. This value can be either your [Pub/Sub subscription ID](https://docs.cloud.google.com/pubsub/docs/create-subscription) or your [fully qualified subscription name](https://www.google.com/url?q=https://docs.cloud.google.com/pubsub/docs/pubsub-basics%23resource_names\&sa=D\&source=docs\&ust=1765569146016537\&usg=AOvVaw3gxdFgpPaceHZPqRl3eZsC). However, if you don't use a fully qualified subscription name, you must also set the `project_id` parameter. | *none* | | **Project ID** | `project_id` | Your [Google Cloud project ID](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects). This parameter is required if the value of `subscription_id` isn't a fully qualified subscription name. | *none* | ### Advanced | Name | Key | Description | Default | | ------ | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | *none* | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry).

In addition to this parameter, the subscription client for the Google Cloud PubSub Input source plugin has its own [queue limit](#queue-limit). | *none* | ## Authentication Use one of the following methods to authenticate with Google Cloud. ### Use GKE workload identity Use this method to provide credentials to ADC through [an attached service account](https://docs.cloud.google.com/docs/authentication/application-default-credentials#attached-sa). This method is supported only if you installed Telemetry Pipeline in a GKE cluster. Additionally, this method assumes that Telemetry Pipeline is installed in the same GCP project as your Pub/Sub topic subscription. 1. Identify the Kubernetes namespace and service account associated with your [Core Instance](/ingest/pipeline/v2/concepts#core-operators-and-core-instances). 2. In Google Cloud, create an IAM service account, or identify an existing IAM service account to associate with your Kubernetes service account. 3. Grant your IAM service account the `roles/pubsub.subscriber` role: ```shell theme={null} gcloud projects add-iam-policy-binding IAM_SA_PROJECT_ID \ --member "serviceAccount:IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com" \ --role "roles/pubsub.subscriber" ``` Replace the following values: * `IAM_SA_PROJECT_ID`: The Google Cloud project ID associated with your IAM service account. * `IAM_SA_NAME`: The name of your IAM service account. 4. Create an IAM allow policy that gives your Kubernetes service account access to impersonate the IAM service account: ```shell theme={null} gcloud iam service-accounts add-iam-policy-binding IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:IAM_SA_PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]" ``` Replace the following values: * `IAM_SA_PROJECT_ID:` The Google Cloud project ID associated with your IAM service account. * `IAM_SA_NAME:` The name of your IAM service account. * `NAMESPACE`: The Kubernetes namespace associated with your pipeline. * `KSA_NAME`: The Kubernetes service account associated with your pipeline. 5. Annotate your Kubernetes service account with the name and project ID of your IAM policy: ```shell theme={null} kubectl annotate serviceaccount KSA_NAME \ --namespace NAMESPACE \ iam.gke.io/gcp-service-account=IAM_SA_NAME@IAM_SA_PROJECT_ID.iam.gserviceaccount.com ``` Replace the following values: * `IAM_SA_PROJECT_ID:` The Google Cloud project ID associated with your IAM service account. * `IAM_SA_NAME:` The name of your IAM service account. * `NAMESPACE`: The Kubernetes namespace associated with your pipeline. * `KSA_NAME`: The Kubernetes service account associated with your pipeline. 6. In Pipeline CLI, use the `--service-account` flag when you create or update a pipeline that contains this plugin: ```shell theme={null} calyptia {create|update} pipeline --service-account KSA_NAME ``` Replace `KSA_NAME` with the Kubernetes service account associated with your pipeline. ### Use GCP service account credentials Use this method to provide credentials to ADC through [the `GOOGLE_APPLICATION_CREDENTIALS` environment variable](https://docs.cloud.google.com/docs/authentication/application-default-credentials#GAC). This method is supported for all Telemetry Pipeline installation methods, including Telemetry Pipeline installations that aren't running in a GKE cluster. 1. If you don't already have an IAM service account in GCP, create a new IAM service account. 2. [Create a new IAM service account key](https://docs.cloud.google.com/iam/docs/keys-create-delete) to use with Telemetry Pipeline. 3. Download the IAM key that you created. This download should be a JSON file. 4. In your GKE cluster, create a Kubernetes secret from the JSON file that contains your IAM key. 5. Use the `--secret-volume-mount` flag when you create or update the pipeline that contains this plugin: ```shell theme={null} calyptia {create|update} pipeline --secret-volume-mount SECRET:KEY=GOOGLE_APPLICATION_CREDENTIALS ``` Replace the following values: * `SECRET`: The name of the secret you created. * `KEY`: The key that contains your credentials in that secret. Don't replace `GOOGLE_APPLICATION_CREDENTIALS` with any other value. ## Queue limit The subscription client for the GCP PubSub source plugin has a 300 MB queue limit. This means the plugin will pause fetching data from Google Pub/Sub if its subscription queue exceeds 300 MB. If other constraints or limits are placed on your pipeline, the GCP PubSub source plugin might stop fetching data before this queue limit is reached. For example, if you use `mem_buf_limit` or `resources.storage.backlogMemLimit` to enforce a memory limit for a v2 pipeline, and that memory limit is reached, those values take precedence over the 300 MB queue limit. # Google Workspace Audit Logs source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/gcp-workspace Use the Google Workspace Audit Logs source plugin. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Google Workspace Audit Logs [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `gsuite-reporter`) lets you retrieve data from Google Workspace and ingest it into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support the use of a [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the Pipeline Builder interface. This plugin doesn't support duplicates of itself within the same pipeline. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### API Key | Name | Key | Description | Default | | ----------- | -------------- | --------------------------------------------------------------- | ------- | | **API Key** | `access_token` | Either an API key or a service account file for authentication. | *none* | ### Options | Name | Key | Description | Default | | ----------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | | **Enable Telemetry** | `telemetry` | If `true`, enables telemetry through the Google API. Accepted values: `true`, `false`. | `false` | | **Pull Interval** | `pull_interval` | The time between calls to the Google API. Minimum of `1s`. | `30s` | | **Data Directory** | `data_dir` | The storage path to allow resuming data collection. | `/data/storage/gsuite-reporter` | | **Application Name** | `application_name` | The application name you want to get reports from. See [ApplicationName](https://developers.google.com/admin-sdk/reports/reference/rest/v1/activities/list#ApplicationName). | `admin` | | **User Key** | `user_key` | The user you want to get reports from. | `all` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # HTTP source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/http Configure the HTTP source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The HTTP [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http`) lets you collect data from various sources and ingest them into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------- | ------ | ---------------------------------------------------------------- | ------- | | **Port** | `port` | Required. The TCP port used for listening for incoming messages. | `9880` | ### Advanced | Name | Key | Description | Default | | ---------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Tag Key** | `tag_key` | Specifies the key name to overwrite a tag. If set, the value of that key overwrites the tag. | *none* | | **Buffer Max Size** | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin drops that record. | `4M` | | **Buffer Chunk Size** | `buffer_chunk_size` | Sets the default [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin temporarily increases the chunk size up to the value of `buffer_max_size` to accommodate it. | `512k` | | **Successful Response Code** | `successful_response_code` | Sets the code returned by successful responses. Accepted values: `200`, `201`, `204`. | `200` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | # HTTP API collector source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/http-collector Receive pipeline data from the HTTP API collector. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The HTTP API Collector [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http_loader`, alias: `http_alias_collector`) lets you ingest data from your HTTP endpoints into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support duplicates of itself within the same pipeline. Additionally, this plugin can't be used in combination with these source plugins within the same telemetry pipeline: * [Absolute](/ingest/pipeline/plugins/source-plugins/absolute) * [Mandiant ASM](/ingest/pipeline/plugins/source-plugins/mandiant-asm) * [Okta](/ingest/pipeline/plugins/source-plugins/okta) * [Signal Sciences](/ingest/pipeline/plugins/source-plugins/signal-sciences) * [Slack](/ingest/pipeline/plugins/source-plugins/slack) * [Vectra M365](/ingest/pipeline/plugins/source-plugins/vectram365) ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | | **URL** | `url` | Required. The request URL. Supports templating. | *none* | | **HTTP Method** | `method` | The request method. Defaults to `GET`, or `POST` if `body` is set. Supports templating. | `GET` | | **Headers** | `header` | Any request headers, formatted as strings separated by the new line character `\n`. Supports templating. | `User-Agent: Fluent-Bit HTTP Loader Plugin` | | **Optional Request body** | `body` | The request body. Supports templating. | *none* | | **Optional Request body** | `store_response_body` | What to store as the response body. This is meant for large response payloads so the plugin doesn't exceed the 5 MiB limit available for data storage in the Cloud at the pipeline metadata API. Supports templating. | `toJson .Response.Body` | | **Request Timeout** | `timeout` | Controls the request timeout, string duration. If set, must be greater than `0s`. | `0s` | | **Pull Interval** | `pull_interval` | Controls the time between requests, string duration. If set, must be greater than `0s`. Supports templating. | `1s` | | **Wait** | `wait` | How much time to wait before starting collection. Useful to sync with `pull_interval`. Supports templating. Evaluate to a string duration. | `0s` | | **Retry** | `retry` | Controls whether to retry the current request. Supports templating. Evaluate to a Boolean. | `false` | | **Max Retries** | `max_retries` | The maximum number of retries. | `1` | | **Stop** | `stop` | Controls when to stop collecting. Supports templating. | `false` | ### Cookie-Based Authentication | Name | Key | Description | Default | | ---------------------- | -------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------- | | **Auth Cookie URL** | `auth_cookie_url` | Cookie-based authentication URL. | *none* | | **Auth Cookie Method** | `auth_cookie_method` | Cookie-based authentication request method. | Defaults to `GET`, or `POST` if `auth_cookie_body` is set. | | **Auth Cookie Header** | `auth_cookie_header` | Cookie-based authentication request headers. String separated by new line character `\n`. | *none* | | **Auth Cookie Body** | `auth_cookie_body` | Cookie-based authentication request body. | *none* | ### OAuth2 | Name | Key | Description | Default | | ---------------------------- | ------------------------ | ---------------------------------------------------------------------------- | ------- | | **OAuth2 Token URL** | `oauth2_token_url` | OAuth 2.0 token URL. To include multiple URLs, separate each URL by a space. | *none* | | **OAuth2 Client ID** | `oauth2_client_id` | OAuth 2.0 client ID. | *none* | | **OAuth2 Client Secret** | `oauth2_client_secret` | OAuth 2.0 client secret. Sensible field, prefer using pipeline secrets. | *none* | | **OAuth2 Scopes** | `oauth2_scopes` | OAuth 2.0 scopes. String, each scope separated by space. | *none* | | **OAuth2 Additional Params** | `oauth2_endpoint_params` | OAuth 2.0 endpoint parameters. String in URL query string format. | *none* | ### Digest Auth | Name | Key | Description | Default | | ------------ | ---------------------- | ---------------------------------------- | ------- | | **Username** | `auth_digest_username` | Username for HTTP Digest authentication. | *none* | | **Password** | `auth_digest_password` | Password for HTTP Digest authentication. | *none* | ### Storage | Name | Key | Description | Default | | ------------------------ | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | **Data Directory** | `data_dir` | Controls where to store data used to resume collecting. Defaults to `/data/storage` if exists, or a temporary directory if available, otherwise storage is disabled. | `/data/storage` | | **Data Expiration Time** | `data_exp` | Controls how much time data can be used after resume. | `0s` | ### Output | Name | Key | Description | Default | | -------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | **Skip** | `skip` | Controls when to skip sending records to Telemetry Pipeline. Supports templating. Defaults to ignore error status codes, and empty response body. | `{{or (ge .Response.StatusCode 400) (empty .Response.Body)}}` | | **Out** | `out` | Controls what to send to Telemetry Pipeline. Supports templating. Defaults to send the response body. | `{{toJson .Response.Body}}` | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ## Templating The available [go-templates](https://pkg.go.dev/text/template) include the following data: * `Index`: (`int`) always available. Is the current fetch index. * `Request`: (`Request` struct) available after a successful fetch. * `Response`: (`Response` struct) available after a successful fetch. * `LastRequestTime` (`*time.Time`) stores the time when the last request was made. Available after a successful request. * `LastResponseTime` (`*time.Time`) stores the time when the last response was received. Available after a successful request and response roundtrip. The `Request` struct includes these fields: * `Method`: `string`. * `URL`: `*url.URL`. * `Header`: `http.Header`. * `Body`: `any`. The `Response` struct includes these fields: * `StatusCode`: `int`. * `Header`: `http.Header`. * `Body`: `any`. You can also use any of the following functions provided by [sprig](http://masterminds.github.io/sprig/): * `timeRFC3339`: `func() string` returns a constant RFC3339 (`2006-01-02T15:04:05Z07:00`) time format. Can be useful to format `time.Time` as JSON timestamp. * `nextLink`: `func (http.Header) string` returns the URL from the header `Link: rel=next`. Useful for pagination. * `parseDuration`: `func (string) (Duration, error)` parses a string as `time.Duration`. * `has`: `func(obj any, key string)` checks whether the given key exists inside an object. * `jq`: `func(query string, data any) (any, error)` transforms data using `jq`. * `log`: `func(any...)` prints a message to stdout. * `logf`: `func(format string, args any...)` prints a formatted message to stdout. * `set_variable`: `func(key string, value any)` store a persistent variable you can reference later on inside other go-template executions. * `has_variable`: `func(key string)` checks whether the given variable is set. * `get_variable`: `func(key string)` retrieve a previously stored persistent variable, returns the value or an empty string. * `unset_variable`: `func(key string)` deletes a previously stored persistent variable. ### Time A common action is to manipulate time parameters. Here are some common operations: * `now` returns the current time as `time.Time` type. * `now.Format "2006-01-02T15:04:05Z07:00"` returns the current time as a string in RFC3339 format. * `now.Format timeRFC3339` same as before but using a utility constant. * `mustToDate timeRFC3339 .Response.Body.time` converts the `time` field inside the response body to `time.Time` using the RFC3339 time format. * `.Response.Body.time | mustToDate timeRFC3339` same as before but using the pipe operator. * `mustDateModify "+1h" now` return the current time plus 1 hour as `time.Time`. * `now | mustDateModify "-1h"` similar to the previous one, but uses the pipe operator, and also subtracts 1 hour. - `(now | mustDateModify "+1h").Format timeRFC3339` returns the current time plus 1 hour as a string in RFC3339 time format. Notice the usage of parenthesis. - `((.Response.Body.time | mustToDate timeRFC3339) | mustDateModify "+1h").Format timeRFC3339` parses the response body `time` field to `time.Time` then adds 1 hour, then formats it to string in RFC3339 time format. ### Additional Go template resources * [Go package template documentation](https://pkg.go.dev/text/template) * Gopher Academy Blog [Using Go Templates](https://blog.gopheracademy.com/advent-2017/using-go-templates/) post * [Mastering HTML templates in Go - The fundamentals](https://philipptanlak.com/mastering-html-templates-in-go-the-fundamentals/), by Philipp Tanlak * HashiCorp [Learn Go template syntax](https://developer.hashicorp.com/nomad/tutorials/templates/go-template-syntax) page * [Sprig Function Documentation](http://masterminds.github.io/sprig/) ## Examples Use the following examples to help you retrieve data from your HTTP endpoints and ingest it into Chronosphere Telemetry Pipeline. ### SWAPI example Here's an example using the [SWAPI](https://swapi.dev) API: ```yaml theme={null} pipeline: inputs: - name: http_loader url: |- {{with .Response.Body.next}}{{.}}{{else}}https://swapi.dev/api/people{{end}} out: |- {{toJson .Response.Body.results}} skip: |- {{or (ge .Response.StatusCode 400) (empty .Response.Body.results)}} ``` By fetching `https://swapi.dev/api/people`, you'll receive a response body similar to the following: ```json theme={null} { "next": "https://swapi.dev/api/people/?page=2", "results": [ {...}, {...} ] } ``` #### URL If a value is returned for `next` in the response body, be sure to use that as the URL. Otherwise, set the default to `https://swapi.dev/api/people`. #### Out Select the `results` field from the response body and transform it to JSON. This will attempt to send an array to Telemetry Pipeline. However, arrays aren't supported, which causes the plugin to split the array and send each item in the following manner: ```json theme={null} { "index": 1, "value": {} } ``` ```json theme={null} { "index": 2, "value": {} } ``` ```json theme={null} { "index": 3, "value": {} } ``` Another way to visualize this is: ```js theme={null} arr.map((value, index) => ({ index, value })).forEach(send) ``` #### Skip With skip, you can control when to skip sending data to Telemetry Pipeline. As an example, you can skip if the response status code is greater or equal to 400, or if the `results` field in the response body is empty. #### Transform Say you are only interested in the `name` property on each result. ```json theme={null} { "next": "https://swapi.dev/api/people/?page=2", "results": [ { "name": "Luke Skywalker", // ... }, { "name": "C-3PO", // ... } ] } ``` Inside the go-templating, you can use `jq` to transform the data. Change `out` to the following to map over each result and extract only the `name`: ```handlebars theme={null} {{.Response.Body | jq ".results.[] | {name}" | toJson}} ``` ### JSON placeholder example The [JSON placeholder](https://jsonplaceholder.typicode.com) API supports pagination, but doesn't provide a `next` URL to lookup. Instead, the client should pass a `_page` query string parameter, starting at `1`: ```yaml theme={null} pipeline: inputs: - name: http_loader url: |- https://jsonplaceholder.typicode.com/posts?_limit=10&_page={{add .Index 1}} retry: |- {{ge .Response.StatusCode 500}} max_retries: 3 ``` #### Index Inside the template context you will always find the variable `.Index`, which is an auto-increasing number each time the plugin makes a request. It starts at `0`. In this case, to start the pagination at `1`, add `1` to the index to advance the page. #### Retry You can enable retrying of requests. In this case, you want to retry if you get a response with an status code greater than or equal to `500`. Also, in the case the HTTP client fails for some other reason (perhaps a networking issue) and you don't get a response back, the request will be retried. You can control how many tries will be performed using `max_retries`. In this case, the plugin will try at most three (`3`) times. ### Okta example When using the [Okta API](https://developer.okta.com/docs/reference/api/system-log/), it returns a `Link: ; rel="next"` header which you can use for pagination: ```yaml theme={null} pipeline: inputs: - name: http_loader url: |- {{- with nextLink .Response.Header -}} {{.}} {{- else -}} https://{replaceWithYourDomain}/api/v1/logs {{- end -}} header: "Authorization: SSWS {{secrets.oktaAPIToken}}" ``` #### Next link There's a helper function called `nextLink` that takes some headers, and then finds the `Link` header with `rel=next`. In case you get the header, use that as the URL, otherwise you'll set the default Okta URL of your domain. Notice that `with` works as an `if`, but overrides that current template data with the result. In this case, it's equivalent to: ```handlebars theme={null} {{- if nextLink .Response.Header -}} {{nextLink .Response.Header}} ``` ### Carbon Black example The [Carbon Black](https://developer.carbonblack.com/reference/carbon-black-cloud/platform-apis/) API uses an offset style pagination controlled by `start` and `rows`. You should be more interested in the time-range filtering, and mixing pagination with it. Control time range with either a combination of `time_range.start` and `time_range.end`, which are both fixed timestamps. There's also the option `range` a relative value (`-2h`). However, to have complete control, use fixed timestamps. Issue a `POST` request with a body like the following: ```json theme={null} { "time_range": { "start": "2023-08-14T01:00:00.000Z", "end": "2023-08-14T02:00:00.000Z" }, "start": 1, "rows": 5 } ``` Advance the page by changing `start` to previous `start` + `rows`. For example, `1`, `6`, `11`, `16`, and so on. Change `time_range` only after you have finished paging through the current time range. ```yaml theme={null} pipeline: inputs: - name: http_loader url: https://defense.conferdeploy.net/api/alerts/v7/orgs/ABCD1234/alerts/_search header: |- Content-Type: application/json X-Auth-Token: {{secrets.blackCarbonAuthToken}} body: |- {{- $now := now.UTC.Truncate (parseDuration "1h") -}} {{- $timeStart := $now | mustDateModify "-1h" -}} {{- $timeEnd := $now -}} {{- $start := 1 -}} {{- if and .Request .Response -}} {{- $prevTimeEnd := .Request.Body.time_range.end | mustToDate timeRFC3339 -}} {{- $start = add .Request.Body.start 5 -}} {{- if or (empty .Response.Body.results) (lt (len .Response.Body.results) 5) -}} {{- $timeStart = $prevTimeEnd -}} {{- $timeEnd = ($timeStart | mustDateModify "+1h") -}} {{- $start = 1 -}} {{- end -}} {{- if $timeEnd.Before $prevTimeEnd -}} {{- $timeEnd = $prevTimeEnd -}} {{- $timeStart = ($timeEnd | mustDateModify "-1h") -}} {{- end -}} {{- end -}} {"time_range":{"start":"{{$timeStart.Format timeRFC3339}}","end":"{{$timeEnd.Format timeRFC3339}}"},"start":{{$start}},"rows":5} out: |- {{toJson .Response.Body.results}} skip: |- {{or (ge .Response.StatusCode 400) (empty .Response.Body.results)}} pull_interval: |- {{- if ge .Response.StatusCode 400 -}} 10s {{- else -}} {{- $interval := parseDuration "100ms" -}} {{- if or (empty .Response.Body.results) (lt (len .Response.Body.results) 5) -}} {{- $interval = parseDuration "1h" -}} {{- $currentTimeEnd := .Request.Body.time_range.end | mustToDate timeRFC3339 -}} {{- $nextTimeEnd := $currentTimeEnd.Add $interval -}} {{- if $nextTimeEnd.After now -}} {{- $interval = $nextTimeEnd.Sub now -}} {{- if le $interval.Nanoseconds 0 -}} {{- $interval = parseDuration "1ns" -}} {{- end -}} {{- end -}} {{- end -}} {{$interval}} {{- end -}} ``` Make use of go-templating variables to keep order. #### Body In the `body`, first define a variable to hold the current timestamp truncated to a 1 hour unit. Use it to define both time range start and end. Also, set the pagination to `start` at `1`. The variables `.Request` and `.Response` are not initially available, and are available only after a successful fetch. The following code happens inside an `if` that determines if both of them are available. There you can retrieve the previous time range end. Advanced the page by increasing the pagination `start` by `5` (rows). Then, determine if less than `5` (rows) `results` were returned, which means you have reached the pages end, and pagination is finished. In this case, change the time range window by adding to the previous request values. You can reset pagination by setting page `start` to `1`. There's an additional `if` that checks if the time range window is less than the previous time range window, in which case it sets it to the previous one. Finally, use these variables to construct the JSON request body. #### Pull interval Then, it's important to set `pull_interval`. If you're not careful, you could start to ask for time windows in the future, which wouldn't return any data. First, do a validation check. If you receive a status code greater than or equal to 400, return `10s` as `pull_interval` (you can even use a higher value). Now, set an initial pull interval of `100ms`. Then, in case `results` inside the response body is empty or less than 5 (rows), it means you have finished paginating. In such case, move the time window. To do so, increase the `pull_interval` to `1h`. Be sure to add an `if` to avoid moving the time window into the future, in which case you can limit the pull interval to `next - now`. Determine if the resulting `pull_interval` is negative, and set it to a minimum value of `1ns`. In resume, in case of error, use a pull interval of `10s`. In case the pagination has ended, use `1h` but maxed to now. Otherwise, if you're still paginating, set it to `100ms`. ### Sentinel One example When using the [Sentinel One](https://www.postman.com/api-evangelist/workspace/sentinelone/overview) API, pagination uses a cursor-based style. You can also control a time window, similar to how it is done on the Carbon Black API. The Carbon Black API used a token inside the request headers. But, because Sentinel One uses cookies, there's a separate login endpoint that you can specify in the plugin to fetch the auth cookie, and then store in the cookie jar. ```yaml theme={null} pipeline: inputs: - name: http_loader url: |- {{- $now := now.UTC.Truncate (parseDuration "1h") -}} {{- $start := $now | mustDateModify "-1h" -}} {{- $end := $now -}} {{- if and .Request .Response -}} {{- $prevEnd := .Request.URL.Query.Get "updatedAt__lte" | mustToDate timeRFC3339 -}} {{- if (empty .Response.Body.pagination.nextCursor) -}} {{- $start = $prevEnd -}} {{- $end = ($start | mustDateModify "+1h") -}} {{- end -}} {{- if $end.Before $prevEnd -}} {{- $end = $prevEnd -}} {{- $start = ($end | mustDateModify "-1h") -}} {{- end -}} {{- end -}} https://usea1-partners.sentinelone.net/web/api/v2.1/threats?updatedAt__gte={{$start.Format timeRFC3339}}&updatedAt__lte={{$end.Format timeRFC3339}}&limit=5{{with .Response.Body.pagination.nextCursor}}&cursor={{.}}{{end}} out: |- {{toJson .Response.Body.data}} skip: |- {{or (ge .Response.StatusCode 400) (not (empty .Response.Body.errors)) (empty .Response.Body.data)}} pull_interval: |- {{- if or (ge .Response.StatusCode 400) (not (empty .Response.Body.errors)) -}} 10s {{- else -}} {{- $interval := parseDuration "100ms" -}} {{- if empty .Response.Body.pagination.nextCursor -}} {{- $interval = parseDuration "1h" -}} {{- $currentEnd := .Request.URL.Query.Get "updatedAt__lte" | mustToDate timeRFC3339 -}} {{- $nextEnd := $currentEnd.Add $interval -}} {{- if $nextEnd.After now -}} {{- $interval = $nextEnd.Sub now -}} {{- if le $interval.Nanoseconds 0 -}} {{- $interval = parseDuration "1ns" -}} {{- end -}} {{- end -}} {{- end -}} {{$interval}} {{- end -}} auth_cookie_url: |- https://usea1-partners.sentinelone.net/web/api/v2.1/users/login/by-api-token auth_cookie_body: |- {"data": {"apiToken": "{{secrets.sentinelOneAPIToken}}"}} auth_cookie_header: |- Content-Type: application/json ``` #### Auth Cookie URL If you set `auth_cookie_url`, the plugin issues a request to this URL before starting collecting. #### Auth Cookie Exp You can control for how much time the cookie is valid by using `auth_cookie_exp`, which accepts a duration string. After the cookie is expired, the plugin will attempt to renew the cookie. This body doesn't support go-templates. The notation `{{ secrets.* }}` is a feature available in Telemetry Pipeline. Its syntax might resemble go-templates, but they are not go-templates. ### Dynatrace example The following examples uses the [Dynatrace API](https://www.dynatrace.com/support/help/dynatrace-api/environment-api/audit-logs/get-log). One important aspect of this API is that it uses a `nextPageKey` for pagination on the URL query string parameters, but this parameter is exclusive. When setting this parameter, you can set only that parameter, and not the rest of the filters. This prevents you from referencing the previous request parameters. For that, use the `has_variable`, `get_variable`, `set_variable` and `unset_variable` helpers so you can persist data in the plugin execution. These variables are persisted even after the Telemetry Pipeline process is restarted. ```yaml theme={null} pipeline: inputs: - name: http_loader url: |- {{- $environmentID := "ggy66547" -}} {{- with .Response.Body.nextPageKey -}} https://{{$environmentID}}.live.dynatrace.com/api/v2/auditlogs?nextPageKey={{.}} {{- else -}} {{- $now := now.UTC.Truncate (parseDuration "1h") -}} {{- $from := $now | mustDateModify "-1h" -}} {{- $to := $now -}} {{- if has_variable "to" -}} {{- $prevTo := get_variable "to" | mustToDate timeRFC3339 -}} {{- $from = $prevTo -}} {{- $to = ($from | mustDateModify "+1h") -}} {{- if $to.Before $prevTo -}} {{- $to = $prevTo -}} {{- $from = ($to | mustDateModify "-1h") -}} {{- end -}} {{- end -}} {{- set_variable "to" ($to.Format timeRFC3339) -}} https://{{$environmentID}}.live.dynatrace.com/api/v2/auditlogs?from={{$from.Format timeRFC3339}}&to={{$to.Format timeRFC3339}}&sort=timestamp&pageSize=10 {{- end -}} header: "Authorization: Api-Token {{secrets.dynatraceAPIToken}}" out: |- {{toJson .Response.Body.auditLogs}} skip: |- {{or (ge .Response.StatusCode 400) (empty .Response.Body.auditLogs)}} pull_interval: |- {{- if ge .Response.StatusCode 400 -}} {{ log "error:" .Response.Body }} 10s {{- else -}} {{- $interval := parseDuration "100ms" -}} {{- if empty .Response.Body.nextPageKey -}} {{- $interval = parseDuration "1h" -}} {{- $currentTo := get_variable "to" | mustToDate timeRFC3339 -}} {{- $nextTo := $currentTo.Add $interval -}} {{- if $nextTo.After now -}} {{- $interval = $nextTo.Sub now -}} {{- if le $interval.Nanoseconds 0 -}} {{- $interval = parseDuration "1ns" -}} {{- end -}} {{- end -}} {{- end -}} {{$interval}} {{- end -}} ``` The `url` go-template determines if you received a `nextPageKey` in the response body. In that case, use that to paginate. Otherwise, set the time window parameters, similar to the previous examples, but also make use of `set_variable` to persist the `to` parameter. Doing that lets you reference it later, and not depend on the previous request. # Microsoft Intune - Audit event source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/intune Ingest Microsoft Intune - Audit events in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Microsoft Intune - Audit event [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http_loader`, alias: `microsoft-intune-audit-events`) lets you ingest data from Microsoft Intune logs into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support the use of a [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the Pipeline Builder interface. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------ | ---------------------- | ------------------------------------------------- | ------- | | **Tenant ID** | `tenantID` | Required. Your Microsoft tenant ID. | *none* | | **OAuth2 Client ID** | `oauth2_client_id` | Required. Your Microsoft OAuth 2.0 client ID. | *none* | | **OAuth2 Client Secret** | `oauth2_client_secret` | Required. Your Microsoft OAuth 2.0 client secret. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # Kafka source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/kafka Configure the Kafka source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. You can use the Kafka [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `kafka`, alias: `kafka_input`) to ingest data from your Apache Kafka instances into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------- | --------- | ------------------------------------------------------------------------------------------------------------------------------ | ------- | | **Brokers** | `brokers` | Required. Single of multiple lists of Kafka brokers. For example, `192.168.1.3:9092`, `192.168.1.4:9092`. | *none* | | **Topics** | `topics` | Required. Single entry or list of topics separated by comma (`,`) that Telemetry Pipeline uses to receive messages from Kafka. | *none* | ### Advanced | Name | Key | Description | Default | | --------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Minimum Queued Messages** | `rdkafka.queued.min.messages` | Minimum number of messages per topic and partition Telemetry Pipeline tries to maintain in the local consumer queue. | `10` | | **Group ID** | `rdkafka.group.id` | The client group ID string. All clients with the same group ID belong to the same group. | *none* | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Other This parameter doesn't have an equivalent setting in the Telemetry Pipeline web interface, but you can use it in pipeline configuration files. | Name | Key | Description | Default | | ------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | *none* | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single log exceeds this size, the plugin drops that log. | `4M` | ### Extended librdkafka parameters This plugin uses the [librdkafka](https://github.com/confluentinc/librdkafka) library. Certain configuration parameters available through the Telemetry Pipeline UI are based on librdkafka settings. These parameters generally use the `rdkafka.` prefix. In addition to the parameters available through the Telemetry Pipeline UI, you can customize any of the [librdkafka configuration properties](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) by adding them to a pipeline configuration file. To do so, append the `rdkafka.` prefix to the name of that property. For example, to customize the `socket.keepalive.enable` property, add the `rdkafka.socket.keepalive.enable` key to your configuration file. Don't use librdkafka properties to configure a pipeline's memory buffer. Instead, use the [`buffer_max_size`](#other) parameter. # Kubernetes Events source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/kubernetes Ingest Kubernetes events in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Kubernetes Events [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `kubernetes_events`) lets you ingest data from Kubernetes into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ---------------------------------- | ---------- | -------------------------------------------------------------------------- | -------------------------------- | | **Kubernetes API Server Endpoint** | `kube_url` | Required. The API server endpoint that's accessible to Telemetry Pipeline. | `https://kubernetes.default.svc` | ### Advanced | Name | Key | Description | Default | | ----------------------------------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Kubernetes TLS CA file** | `kube_ca_file` | Your Kubernetes CA certificate file. | *none* | | **Kubernetes TLS CA path** | `kube_ca_path` | Your Kubernetes CA certificate path. | *none* | | **Kubernetes authorization Token File**\* | `kube_token_file` | Your Kubernetes authorization token file. | *none* | | **Kubernetes Token Time To Live (TTL)** | `kube_token_ttl` | The amount of time before a token is reread from its file. | `10M` | | **Kubernetes Request Limit** | `kube_request_limit` | The request limit for how many events to collect. | `0` | | **Kubernetes Event Retention Time** | `kube_retention_time` | How long Kubernetes should retain events. | `1h` | | **Kubernetes Namespace** | `kube_namespace` | The Kubernetes namespace to query events from. By default, fetches events from all namespaces. | *none* | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # Mandiant ASM source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/mandiant-asm Configure the Mandiant ASM source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Mandiant ASM [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http_loader`, alias: `mandiant_asm`) lets you query Mandiant ASM to ingest data into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support the use of a [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the Pipeline Builder interface. This plugin doesn't support duplicates of itself within the same pipeline. Additionally, this plugin can't be used in combination with these source plugins within the same telemetry pipeline: * [Absolute](/ingest/pipeline/plugins/source-plugins/absolute) * [HTTP API collector](/ingest/pipeline/plugins/source-plugins/http-collector) * [Okta](/ingest/pipeline/plugins/source-plugins/okta) * [Signal Sciences](/ingest/pipeline/plugins/source-plugins/signal-sciences) * [Slack](/ingest/pipeline/plugins/source-plugins/slack) * [Vectra M365](/ingest/pipeline/plugins/source-plugins/vectram365) ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General The configured fields replace sections of the `url` field in the YAML file in the following way: ```text theme={null} url: https://asm-api.advantage.mandiant.com/api/v1/search/SEARCH_ELEMENT/SEARCH_QUERY ``` | Name | Key | Description | Default | | ------------------ | --------------------- | ------------------------------------------------------------------------------------- | ---------- | | **Project ID** | `PROJECT_ID` | Required. | *none* | | **Search element** | *none* | Required. Used in `url` field. Accepted values: `entities`, `issues`, `technologies`. | `entities` | | **Search Query** | *none* | Required. Used in `url` field. | *none* | | **Access Key** | `INTRIGUE_ACCESS_KEY` | Required. | *none* | | **Secret Key** | `INTRIGUE_SECRET_KEY` | Required. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # Mock Data source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/mock-data Configure the Mock Data source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Mock Data [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `dummy`) lets you input user-generated data into a telemetry pipeline. Although this is a self-contained plugin that doesn't communicate with external sources, it's still classified as a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. To use the Mock Data source plugin, you must create your own dataset and feed that data to the plugin. If you'd prefer to have Chronosphere Telemetry Pipeline generate simulated log data on your behalf, use the [Datagen](/ingest/pipeline/plugins/source-plugins/datagen) plugin instead. This plugin doesn't support duplicates of itself within the same pipeline. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ------------- | ------- | ----------------------------------------- | ------- | | **Mock Data** | `dummy` | Required. Your mock data, in JSON format. | *none* | ### Advanced | Name | Key | Description | Default | | ---------------------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Rate** | `rate` | The amount of events per second to mock. | `1` | | **Samples** | `samples` | How many times to generate events. | `0` | | **Base Timestamp in seconds** | `start_time_sec` | A dummy base timestamp, in seconds. | `-1` | | **Base Timestamp in nano seconds** | `start_time_nsec` | A dummy base timestamp, in nanoseconds | `-1` | | **Enable Fixed Timestamp** | `fixed_timestamp` | If `true`, use a fixed timestamp, which lets the message pre-generate once. Accepted values: `true`, `false` | `false` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # Okta System Logs Collector source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/okta Receive pipeline data from Okta System Logs Collector. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Okta System Logs Collector [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http_loader`, alias: `okta_system_logs`) lets you ingest data from your Okta instances into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support the use of a [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the Pipeline Builder interface. This plugin doesn't support duplicates of itself within the same pipeline. Additionally, this plugin can't be used in combination with these source plugins within the same telemetry pipeline: * [Absolute](/ingest/pipeline/plugins/source-plugins/absolute) * [HTTP API collector](/ingest/pipeline/plugins/source-plugins/http-collector) * [Mandiant ASM](/ingest/pipeline/plugins/source-plugins/mandiant-asm) * [Signal Sciences](/ingest/pipeline/plugins/source-plugins/signal-sciences) * [Slack](/ingest/pipeline/plugins/source-plugins/slack) * [Vectra M365](/ingest/pipeline/plugins/source-plugins/vectram365) ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------- | --------------- | --------------------------------------------------- | ------- | | **Okta Domain URL** | `url` | The URL of your Okta domain. | *none* | | **Okta API Token** | `api_token` | Your Okta API token. | *none* | | **Pull Interval** | `pull_interval` | The time between each API call in Okta system logs. | `10s` | ### Advanced | Name | Key | Description | Default | | ------------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | **Optional Request body** | `body` | Optional request body. For example, `{"custom":"data"}`. | *none* | | **Request Timeout** | `timeout` | The amount of time between requests. | `10s` | | **Max Response Bytes** | `max_response_bytes` | Whether to limit the amount of bytes to read from the response body. | `1572860` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### OAuth2 | Name | Key | Description | Default | | ---------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **OAuth2 Client ID** | `oauth2_client_id` | Optional OAuth 2.0 Client ID. You need to at least pass the client ID, secret and token URL to enable OAuth 2.0 authentication. Uses the `client_credentials` flow. | *none* | | **OAuth2 Client Secret** | `oauth2_client_secret` | Optional OAuth 2.0 client secret. | *none* | | **OAuth2 Token URL** | `oauth2_token_url` | Optional OAuth 2.0 token URL in the formal of HTTP or HTTPS. For example, `https://example.org/oauth2/token`. | *none* | | **OAuth2 Scopes** | `oauth2_scopes` | Optional list of additional scopes for OAuth 2.0 authentication separated by space. For example, `scope1 scope2`. | *none* | | **OAuth2 Scopes Separator** | `oauth2_scopes_separator` | Additional parameter to pass during OAuth 2.0 authentication that separates scopes specified in OAuth 2.0 Scopes. | *none* | | **OAuth2 Additional Params** | `oauth2_endpoint_params` | Optional additional parameters to add during OAuth 2.0 authentication. The format is a URL query string. For example, `foo=bar&bar=qux`. | *none* | # OpenTelemetry source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/opentelemetry Configure the OpenTelemetry source plugin in Chronosphere Telemetry Pipeline to ingest logs, metrics, and traces for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The OpenTelemetry [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `opentelemetry`) lets you ingest data from your OpenTelemetry instances into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :--------------------------: | :--------------------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | -------- | ------ | ---------------------------------------------------------------- | ------- | | **Port** | `port` | Required. The TCP port used for listening for incoming messages. | `8088` | ### Advanced | Name | Key | Description | Default | | ---------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Tag from URI** | `tag_from_uri` | Sets tag from URI. For example, `v1_metrics` from /v1/metrics. | `false` | | **Tag Key** | `tag_key` | Specifies the key name of a tag to overwrite. If set, the tag name will be overwritten by the specified tag key. | *none* | | **Chunk Size** | `buffer_chunk_size` | Sets the default [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin temporarily increases the chunk size up to the value of `buffer_max_size` to accommodate it. | `512K` | | **Buffer Max Size** | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin drops that record. | `4M` | | **Successful Response Code** | `successful_response_code` | Sets the code returned by successful responses. Accepted values: `200`, `201`, `204`. | `201` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | # Prometheus Remote Write source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/prometheus-remote Configure the Prometheus Remote Write source plugin in Chronosphere Telemetry Pipeline to ingest metrics for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Prometheus Remote Write [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `prometheus_remote_write`) lets you ingest data from the [Prometheus Remote-Write API](https://prometheus.io/docs/specs/remote_write_spec/) into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :-----------------: | :--------------------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | -------- | ------ | ---------------------------------------------------- | ------- | | **Port** | `port` | Required. The port for the pipeline to receive data. | `8088` | ### Advanced | Name | Key | Description | Default | | ----------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | **Tag from URI** | `tag_from_uri` | Sets tag from URI. For example, v1\_metrics from /v1/metrics. | `false` | | **URI** | `uri` | Sets the URI to receive Prometheus Remote Write metrics on. | `/api/prom/push` | | **Chunk Size** | `buffer_chunk_size` | Sets the default [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin temporarily increases the chunk size up to the value of `buffer_max_size` to accommodate it. | `512K` | | **Buffer Max Size** | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin drops that record. | `4M` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | # Prometheus scrape source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/prometheus-scrape Configure the Prometheus scrape source plugin in Chronosphere Telemetry Pipeline to ingest metrics for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. You can use the Prometheus scrape [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `prometheus_scrape`) to ingest data from your Prometheus instances and into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :-----------------: | :--------------------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ------------------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ---------- | | **Host** | `host` | Required. The host of the Prometheus metric endpoint you want to scrape. | *none* | | **Port** | `port` | Required. The port of the Prometheus metric endpoint you want to scrape. | *none* | | **Metrics URI Endpoint** | `metrics_path` | Required. The metrics URI endpoint, which must start with a forward slash. Parameters can also be added to the path by using \`?\`. | `/metrics` | | **Scrape Interval** | `scrape_interval` | Required. The interval to scrape metrics. | `10s` | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # S3 Input (SQS) source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/s3-sqs Configure the S3 Input (SQS) source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. > Requires [pipeline agent](/ingest/pipeline/v2/component-versions) v25.8.1 or later, > [Core Operator](/ingest/pipeline/v2/component-versions) v3.67.0 or later, and > [Pipeline CLI](/ingest/pipeline/pipeline-cli) v3.66.0 or later. The S3 Input (SQS) [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `s3_sqs`) lets you continuously ingest new data from Amazon S3 buckets into a telemetry pipeline. This plugin monitors an SQS queue configured to receive notifications [directly from S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html) or [through SNS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-subscribe-queue-sns-topic.html), then creates logs from the files described in S3 events. This plugin ignores any SQS notifications that can't be decoded as references to an object in an S3 bucket, but it doesn't [filter](#filtering) notifications by event type. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support duplicates of itself within the same pipeline. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Requirements To use the S3 Input (SQS) plugin, you must meet these requirements: * Your IAM user or IAM role must have the following permissions for the ARN of your SQS queue: * [`sqs:ChangeMessageVisibility`](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html) * [`sqs:DeleteMessage`](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_DeleteMessage.html) * [`sqs:GetQueueAttributes`](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_GetQueueAttributes.html) * [`sqs:GetQueueUrl`](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_GetQueueUrl.html) * [`sqs:ReceiveMessage`](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html) * Your IAM user or IAM role must have the [`s3:GetObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) permission for all buckets configured to notify your SQS queue. * Your SQS queue must have [a redrive policy and a dead letter queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html). Additionally, Chronosphere recommends setting the `maxReceiveCount` of your redrive queue to a value greater than `1`, which lets SQS retry sending messages upon failure. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ------------------------ | ------------------ | ----------------------------------------------------------------------------------------------------------------------- | ------- | | **AWS SQS Queue Name** | `sqs_queue_name` | Required. The name of the SQS queue whose notifications you want to monitor. | *none* | | **AWS SQS Queue Region** | `sqs_queue_region` | Required if `aws_sqs_endpoint` isn't set. The name of the region where your SQS queue exists. For example, `us-east-1`. | *none* | ### Advanced | Name | Key | Description | Default | | :---------------------------------- | :--------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------ | | **Regular Expression Object Match** | `match_regexp` | The regular expression for matching or excluding object keys from S3. This plugin processes notifications only for objects that match the specified regular expression. If not set, the default value of `.*` matches all possible object keys. | `.*` | | **Delete Message from SQS** | `delete_messages` | If `true`, deletes SQS messages after processing the associated S3 data. If `false`, the plugin re-processes each message at an interval specified by your SQS [visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html), and continues to process each message until a [redrive policy](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html) is triggered or until you meet your SQS message retention policy. Chronosphere recommends not modifying this value unless you're testing a pipeline during its initial setup. This is because deleting SQS messages prevents the plugin from processing the same message multiple times and creating duplicate logs. Accepted values: `true`, `false`. | `true` | | **Line Buffer Max Size** | `max_line_buffer_size` | The maximum line size the plugin will read from [JSON or plain text files](#supported-data-types). | `10MiB` | | **S3 Assume Role ARN** | `s3_assume_role_arn` | The ARN of the IAM role for accessing S3 buckets. This can be an ARN within the same account or [across accounts](#cross-account-access). | *none* | | **S3 Role External ID** | `s3_role_external_id` | The [external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html) of the role to assume in S3. | *none* | | **SQS Assume Role ARN** | `sqs_assume_role_arn` | The ARN of the IAM role for accessing the SQS queue. This can be an ARN within the same account or [across accounts](#cross-account-access). | *none* | | **SQS Role External ID** | `sqs_role_external_id` | The [external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html) of the role to assume in SQS. | *none* | | **SQS Queue Owner Account ID** | `sqs_queue_owner_account_id` | The AWS account ID of the queue owner for [cross-account access](#cross-account-access). | *none* | | **S3 Read Concurrency** | `s3_read_concurrency` | The maximum number of concurrent S3 [`GetObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) calls that this plugin will make. | The number of logical CPUs allocated to each pipeline [replica](/ingest/pipeline/v2/configure/scaling). | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ## Authentication methods The S3 Input (SQS) plugin supports the following authentication methods: * [EKS Pod Identities](#eks-pod-identities) * [IMDS](#imds) * [IRSA](#irsa) * [Static credentials](#static-credentials) ### EKS Pod Identities To use EKS Pod Identities for authentication: 1. In AWS, configure [EKS Pod Identities](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html). 2. In [Pipeline CLI](/ingest/pipeline/pipeline-cli), add the following flag to a `create pipeline` or `update pipeline` command: ```shell /VALUE/ theme={null} calyptia {create|update} pipeline --service-account VALUE ``` Replace *`VALUE`* with the name of the Kubernetes service account associated with your Pods. ### IMDS To use IMDS for authentication: * In AWS, configure [IAM roles for your EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html). ### IRSA To use IRSA for authentication: 1. In AWS, [set up IRSA for your EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html). 2. [Assign an IAM role to your Kubernetes service account](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html). 3. In [Pipeline CLI](/ingest/pipeline/pipeline-cli), add the following flag to a `create pipeline` or `update pipeline` command: ```shell /VALUE/ theme={null} calyptia {create|update} pipeline --service-account VALUE ``` Replace *`VALUE`* with the name of your Kubernetes service account. ### Static credentials To use static credentials for authentication: * In Telemetry Pipeline, [create secrets](/ingest/pipeline/v2/configure/secrets#add-a-secret) that contain the values of your [IAM access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). These secrets must use the key names `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. You don't need to add an explicit [reference](/ingest/pipeline/v2/configure/secrets#reference-a-secret) to these secrets in your pipeline configuration file. If secrets with the correct key names are present, the S3 Input (SQS) plugin automatically detects these values and uses them for authentication. ## Cross-account access The S3 Input (SQS) plugin supports cross-account access through the following methods. * [Using IAM roles](#cross-account-access-using-iam-roles) (recommended) * [Using resource-based policies](#cross-account-access-using-resource-based-policies) Cross-account access and authentication are independent. You can use any cross-account access method with any [authentication method](#authentication-methods). ### Cross-account access using IAM roles To set up cross-account access using IAM roles, use the following [configuration parameters](#configuration-parameters): * `sqs_assume_role_arn`: Required. * `sqs_role_external_id`: Required if you need to use an external ID to assume the SQS role. The S3 (SQS) plugin will extract the necessary account ID from the value of `sqs_assume_role_arn`, which means that the `sqs_queue_owner_account_id` parameter isn't required. However, if you do specify a value for `sqs_queue_owner_account_id`, that value takes precedence over the value extracted from `sqs_assume_role_arn`. ### Cross-account access using resource-based policies To set up cross-account access using resource-based policies, use the following [configuration parameters](#configuration-parameters): * `sqs_assume_role_arn`: Required if you're using an assumed role. * `sqs_role_external_id`: Required if you need to use an external ID to assume the SQS role. * `sqs_queue_owner_account_id`: Required if the SQS queue to which your policy is attached has a different owner than the account specified in `sqs_assume_role_arn`, or if you aren't using assumed roles. ## Supported data types The S3 Input (SQS) plugin can ingest [JSON objects](#json) and [plain text](#plain-text) from files stored in S3 buckets, including gzip-compressed files. Additionally, this plugin can extract and ingest compressed and uncompressed files from [tar archives](#tar-archives). ### JSON This plugin can ingest data from JSON files with these file extensions: * `.json` * `.jsonl` * `.ndjson` If a file contains only a single JSON object, this plugin creates a new log from that object. If a file contains multiple [newline-delimited JSON (NDJSON)](https://github.com/ndjson/ndjson-spec) objects, this plugin creates a new log from each JSON object within that file. Key-value pairs from JSON objects are stored as key-value pairs in the resulting log. For JSON files that use gzip compression (with file extensions such as `.json.gzip` or `.json.gz`), this plugin decompresses each file before processing it accordingly. ### Plain text If a file doesn't use a file extension that identifies it as a JSON file, the S3 Input (SQS) plugin processes that file as plain text. It creates a new log from each line of the file and stores the content in a key named `_raw` within the resulting log. For non-JSON files that use gzip compression (with file extensions that include the `.gzip` or `.gz` suffix), this plugin decompresses each file before processing it accordingly. ### Tar archives The plugin can extract and consume files from tar archives with these file extensions: * `.tar` * `.tar.gz` * `.tar.gzip` After the plugin extracts these files, it processes any [JSON](#json) and [plain text](#plain-text) data accordingly, but skips directories and symbolic links. If files inside a tar archive are gzip-compressed, this plugin decompresses those files accordingly. ## Filtering The S3 Input (SQS) plugin doesn't filter notifications by event type. If a notification contains a reference to an object in an S3 bucket, the plugin will ingest data from that object, regardless of its associated event type. To create filters based on event type, you must configure the notification settings of your SQS settings in AWS. For more information, see the AWS [Event notification types and destinations](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html) guide. ## Metadata The S3 Input (SQS) plugin attaches the following metadata to each log: * `__chrono_bucket`: The name of the S3 bucket that contains the file from which the log was created. * `__chrono_file`: The key of the S3 object from which the log was created. * `__chrono_tar_file_entry`: For data extracted from [tar archives](#tar-archives) only. The name of the tar archive that contained the file from which the log was created. ## Get started To get started with the S3 Input (SQS) plugin, follow these steps. 1. Either [create a new pipeline](/ingest/pipeline/v2/build/create-modify#create-a-pipeline) or [modify an existing pipeline](/ingest/pipeline/v2/build/create-modify). 2. For testing purposes, set the pipeline's destination to [standard output](/ingest/pipeline/plugins/destination-plugins/standard-output). 3. Set the pipeline's source to S3 Input (SQS), and then add values for all required parameters, along with any optional parameters of your choosing. 4. Set up one of the supported [authentication methods](#authentication-methods) for the S3 Input (SQS) source plugin. 5. In the Telemetry Pipeline web interface, go to the summary page for that pipeline. 6. In the [**Pipeline Output**](/ingest/pipeline/navigate#pipeline-output-v2-pipelines) section, click **Get latest logs**. 7. Review this log output to ensure that you're receiving data from S3. If you don't receive any data, or if you encounter connection errors, review your plugin configuration settings. 8. After you've confirmed that the S3 Input (SQS) plugin is functioning correctly, you can overwrite the standard output destination with the destination where you want to send your telemetry data. # Signal Sciences source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/signal-sciences Configure the Signal Sciences source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Signal Sciences [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http_loader`) lets you query the Signal Sciences API to ingest data into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support the use of a [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the Pipeline Builder interface. This plugin doesn't support duplicates of itself within the same pipeline. Additionally, this plugin can't be used in combination with these source plugins within the same telemetry pipeline: * [Absolute](/ingest/pipeline/plugins/source-plugins/absolute) * [HTTP API collector](/ingest/pipeline/plugins/source-plugins/http-collector) * [Mandiant ASM](/ingest/pipeline/plugins/source-plugins/mandiant-asm) * [Okta](/ingest/pipeline/plugins/source-plugins/okta) * [Slack](/ingest/pipeline/plugins/source-plugins/slack) * [Vectra M365](/ingest/pipeline/plugins/source-plugins/vectram365) ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General The configured fields replace sections of the `url` field in the YAML file in the following way: ```text theme={null} https://dashboard.signalsciences.net{{with .Response.Body.next.uri}}.{{else}}/api/v0/corps/CORP/sites/SITENAME/requests?q=SEARCH_QUERY{{end}} ``` | Name | Key | Description | Default | | -------------------- | ------ | ------------------------------------------------------------------------- | ------- | | **Corp** | *none* | Required. Added to `url` field. | *none* | | **Site Name** | *none* | Required. Added to `url` field. | *none* | | **Search Query** | *none* | The query to pass to the Signal Sciences API, as part of the `url` field. | *none* | | **API Access Token** | *none* | Your Signal Sciences API access token. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # Slack source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/slack Configure the Slack source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Slack [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http_loader`, alias: `slack`) lets you retrieve data from the Slack API and ingest it into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support the use of a [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the Pipeline Builder interface. This plugin doesn't support duplicates of itself within the same pipeline. Additionally, this plugin can't be used in combination with these source plugins within the same telemetry pipeline: * [Absolute](/ingest/pipeline/plugins/source-plugins/absolute) * [HTTP API collector](/ingest/pipeline/plugins/source-plugins/http-collector) * [Mandiant ASM](/ingest/pipeline/plugins/source-plugins/mandiant-asm) * [Okta](/ingest/pipeline/plugins/source-plugins/okta) * [Signal Sciences](/ingest/pipeline/plugins/source-plugins/signal-sciences) * [Vectra M365](/ingest/pipeline/plugins/source-plugins/vectram365) ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | --------- | ------ | ------------------------------- | ------- | | **Token** | *none* | Required. Your Slack API token. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # Splunk HEC source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/splunk-hec Configure the Splunk HEC source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Splunk HEC [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `splunk`, alias: `splunk_hec`) lets you ingest log data from the Splunk HTTP Event Collector into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------ | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | **Address to Listen On** | `listen` | Required. The address to listen on. | `0.0.0.0` | | **Port** | `port` | Required. The port for Telemetry Pipeline to listen on. | `9880` | | **Tag Key** | `tag_key` | Specifies the key name to overwrite a tag. If set, the value of that key overwrites the tag. | *none* | | **Buffer Max Size** | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin drops that record. | `4M` | | **Buffer Chunk Size** | `buffer_chunk_size` | Required. Sets the default [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin temporarily increases the chunk size up to the value of `buffer_max_size` to accommodate it. | `512K` | | **Splunk Token** | `splunk_token` | Your Splunk token for HTTP HEC. | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # Splunk UF source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/splunk-uf Receive pipeline data from the Splunk Universal Forwarder. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. You can use the Splunk UF [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `tcp`, alias: `Splunk_UF`) to ingest data from your Splunk Universal Forwarder instances into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | -------- | ------ | ---------------------------------------------------------------- | ------- | | **Port** | `port` | Required. The TCP port used for listening for incoming messages. | `5170` | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Format** | `format` | Sets the format. Accepted values: `json`, *none*. | *none* | | **Set Separator** | `separator` | Sets separator. | *none* | | **Chunk Size** | `chunk_size` | Sets the chunk size for incoming messages. | `256kb` | | **Buffer Size** | `buffer_size` | Sets the chunk size for incoming JSON messages. These chunks are then stored and managed in the space available by `buffer_size`. | *none* | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | ## Splunk Universal Forwarder configuration ```text theme={null} [tcpout] defaultGroup = calyptia disabled = false [tcpout:calyptia] server = : sendCookedData = false negotiateProtocolLevel = 0 ``` # SQL DB Input source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/sqldb Use the SQL DB Input source plugin with Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The SQL DB Input [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `sqldb`, alias: `SQL_DB_Input`) lets you ingest log data from a relational database into a telemetry pipeline. If you write your logs to a relational database, you can use the SQL DB Input plugin to convert that data into a pipeline-friendly format. This plugin supports the following SQL formats: * Postgres * MySQL * Oracle * Microsoft SQL Server * SQLite This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support duplicates of itself within the same pipeline. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------- | ---------- | | **DB Type** | `driver` | Required. The SQL driver for your relational data. Accepted values: `postgres`, `mysql`, `oracle`, `sqlserver`, `sqlite`. | `postgres` | | **Database Connection String** | `dsn` | Required. The data source name for your database, in string format. | *none* | | **SQL Query** | `query` | Required. The SQL query to perform. This query supports named arguments in `@named` format. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | **Argument Column** | `columnsForArgs` | A list of column names to turn into arguments, separated by spaces. The most recently scanned row in each specified column becomes a `@named` argument you can use to paginate data. For example, if this key has a value of `id log_time`, the plugin creates two arguments you can use in queries: `@last_id` and `@last_log_time`. | *none* | | **Time Column** | `timeFrom` | The column in your database with the time when each log was ingested. | If unassigned, each ingested log is assigned a time value equivalent to the time the plugin runs its query. | | **Time Format** | `timeFormat` | The time format of your `timeFrom` value. Accepted values: `unix_sec`, `unix_ms`, `unix_us`, `unix_ns`, or any RFC time format layout. | If `timeFrom` is an integer: `unix_ns`. If `timeFrom` is a string: `2006-01-02T15:04:05.999999999Z07:00`. | | **Fetch Interval** | `fetchInterval` | How often to query your database. This value must specify one or more integer and one or more time unit, like `5m` or `1m30s`. Valid time units: `ns`, `us`, `ms`, `s`, `m`, `h`. | `1s` | | **Storage Key** | `storageKey` | A key to store the plugin's metadata, which lets the plugin store arguments and other configuration data even if the plugin restarts. By default, the plugin writes its own metadata in `sqldb_{hash}.gob` format, where `{hash}` is a string generated from your query arguments. You can also specify your own value, which can follow any format and use any file extension. | *none* | | **Storage Path** | `storageDir` | The path to the directory where the plugin writes its data. You can use an absolute or relative path, but Chronosphere recommends using an absolute path. | `/data/storage` (If this directory does not exist, the plugin writes to your operating system's default directory for temporary data.) | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ## Example queries The following examples describe how to configure the SQL DB Input plugin in different hypothetical scenarios. ### Serial ID Given the following database structure, which includes an auto-increasing `id`: ```sql theme={null} CREATE TABLE logs ( id SERIAL PRIMARY KEY, payload JSON NOT NULL, created_at TIMESTAMP NOT NULL DEFAULT now() ); ``` You can use the SQL DB Input plugin to paginate over your data using `id`. To include such a query in your pipeline, configure the plugin as follows: ```yaml theme={null} pipeline: inputs: - name: sqldb dsn: "postgres://user:password@localhost:5432/dbname" query: |- SELECT * FROM logs WHERE @last_id IS NULL OR id > @last_id ORDER BY id LIMIT 100 columnsForArgs: id ``` The previous query sorts by `id`, and then uses the `id` value from the previous query to paginate data. ### Time-based pagination Given the following database structure, which includes a non-sortable `id`: ```sql theme={null} CREATE TABLE logs ( id UUID PRIMARY KEY, payload JSON NOT NULL, created_at TIMESTAMP NOT NULL DEFAULT now() ); ``` You can use a combination of `id` and `created_at` to paginate data. To include such a query in your pipeline, configure the plugin as follows: ```yaml theme={null} pipeline: inputs: - name: sqldb dsn: "postgres://user:password@localhost:5432/dbname" query: |- SELECT * FROM logs WHERE @last_id IS NULL OR @last_created_at IS NULL OR (created_at >= @last_created_at OR (created_at = @last_created_at AND id > @last_id)) ORDER BY created_at, id LIMIT 100 columnsForArgs: "id created_at" ``` The previous query sorts by both `id` and `created_at`, then uses the `id` and `created_at` values from the previous query to paginate data. # Syslog source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/syslog Configure the Syslog source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Syslog [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `syslog`) can collect messages through a Unix socket server using UDP or TCP, or over the network using TCP or UDP. This lets you configure your telemetry pipeline to receive, parse, and process Syslog messages. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | ---------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Port** | `port` | Required. The TCP port used for listening for incoming messages. | `5140` | | **Parser** | `parser` | Specifies an alternative parser for the message. If Mode is set to `tcp` or `udp`, the default parser is `syslog-rfc5424`, otherwise `syslog-rfc3164-local` is used. If your syslog messages have fractional seconds, set this value to `syslog-rfc5424` instead. If you have a custom parser, use Advanced Settings to designate the parser. | *none* | | **Mode** | `mode` | The protocol to use. Accepted values: `tcp`, `udp`. | `tcp` | ### Advanced | Name | Key | Description | Default | | ----------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Buffer Max Size** | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin drops that record. | *none* | | **Buffer Chunk Size** | `buffer_chunk_size` | Sets the default [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin temporarily increases the chunk size up to the value of `buffer_max_size` to accommodate it. | *none* | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | # TCP source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/tcp Configure the TCP source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The TCP [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `tcp`) lets you send custom records to an HTTP endpoint. You can use this plugin to configure your telemetry pipeline to collect data from networked computers. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### Required | Name | Key | Description | Default | | -------- | ------ | ---------------------------------------------------------------- | ------- | | **Port** | `port` | Required. The TCP port used for listening for incoming messages. | `5170` | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Format** | `format` | Sets the format. Accepted values: `json`, *none*. | *none* | | **Separator** | `separator` | Sets separator. | *none* | | **Chunk Size** | `chunk_size` | Sets the chunk size for incoming messages. | *none* | | **Buffer Size** | `buffer_size` | Sets the chunk size for incoming JSON messages. These chunks are then stored and managed in the space available by `buffer_size`. | *none* | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | # Telegraf source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/telegraf Configure the Telegraf source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. You can use the Telegraf [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http`, alias: `Telegraf`) to ingest data from your Telegraf instances into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | -------- | ------ | ---------------------------------------------------------------- | ------- | | **Port** | `port` | Required. The TCP port used for listening for incoming messages. | `9814` | ### Advanced | Name | Key | Description | Default | | ---------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Tag Key** | `tag_key` | Specifies the key name to overwrite a tag. If set, the value of that key overwrites the tag. | *none* | | **Buffer Max Size** | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin drops that record. | `4M` | | **Buffer Chunk Size** | `buffer_chunk_size` | Sets the default [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin temporarily increases the chunk size up to the value of `buffer_max_size` to accommodate it. | `512k` | | **Successful Response Code** | `successful_response_code` | Sets the code returned by successful responses. Accepted values: `200`, `201`, `204`. | `200` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | # Vectra M365 - Incident Detection source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/vectram365 Use the Vectra M365 - Incident Detection source plugin. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Vectra M365 - Incident Detection [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http_loader`, alias: `vectra-m365-incident-detection`) lets you retrieve data from Vectra and ingest it into a telemetry pipeline. This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. This plugin doesn't support the use of a [descriptive metadata name](/ingest/pipeline/plugins#descriptive-names) in the Pipeline Builder interface. This plugin doesn't support duplicates of itself within the same pipeline. Additionally, this plugin can't be used in combination with these source plugins within the same telemetry pipeline: * [Absolute](/ingest/pipeline/plugins/source-plugins/absolute) * [HTTP API collector](/ingest/pipeline/plugins/source-plugins/http-collector) * [Mandiant ASM](/ingest/pipeline/plugins/source-plugins/mandiant-asm) * [Okta](/ingest/pipeline/plugins/source-plugins/okta) * [Signal Sciences](/ingest/pipeline/plugins/source-plugins/signal-sciences) * [Slack](/ingest/pipeline/plugins/source-plugins/slack) ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ------------------------ | ---------------------- | ------------------------------------------------------------------- | ------- | | **Vectra Portal URL** | `vectra_portal_url` | Required. Your Vectra portal URL. | *none* | | **OAuth2 Client ID** | `oauth2_client_id` | Required. Your OAuth 2.0 client ID for accessing the Vectra portal. | *none* | | **OAuth2 Client Secret** | `oauth2_client_secret` | Required. The OAuth 2.0 secret key for accessing the Vectra portal. | *none* | ### Advanced | Name | Key | Description | Default | | ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | # Vercel Logs source plugin Source: https://docs.chronosphere.io/ingest/pipeline/plugins/source-plugins/vercel Configure the Vercel Logs source plugin in Chronosphere Telemetry Pipeline to ingest logs for processing and routing. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The Vercel Logs [source plugin](/ingest/pipeline/plugins/source-plugins) (name: `http`, alias: `vercel_logo`) lets you ingest data from your Vercel instances into a telemetry pipeline. This is a [push-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin. ## Supported telemetry types The for Chronosphere Telemetry Pipeline supports these telemetry types: | Logs | Metrics | Traces | | :--------------------------: | :-----------------: | :-----------------: | | | | | ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. ### General | Name | Key | Description | Default | | ----------------------------- | ---------------- | ---------------------------------------------------------------- | -------------------------------------------------------- | | **Port** | `port` | Required. The TCP port used for listening for incoming messages. | `9880` | | **Vercel Verification Token** | `success_header` | Required. Your Vercel verification token. | `x-vercel-verify {INSERT TOKEN FROM LOG DRAIN SETTINGS}` | ### Advanced | Name | Key | Description | Default | | ---------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Tag Key** | `tag_key` | Specifies the key name to overwrite a tag. If set, the value of that key overwrites the tag. | *none* | | **Buffer Max Size** | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin drops that record. | `4M` | | **Buffer Chunk Size** | `buffer_chunk_size` | Sets the default [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single record exceeds this size, the plugin temporarily increases the chunk size up to the value of `buffer_max_size` to accommodate it. | `512k` | | **Successful Response Code** | `successful_response_code` | Sets the code returned by successful responses. Accepted values: `200`, `201`, `204`. | `200` | | **Memory Buffer Limit** | `mem_buf_limit` | Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced.

For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type. To learn more, see the v2 [backpressure](/ingest/pipeline/v2/configure/backpressure) guide.

For v3 pipelines, this parameter is independent from the [OpenTelemetry `memory_limiter` and `batch` processors](/ingest/pipeline/v3#route-data-from-fluent-bit-to-opentelemetry). | *none* | ### Security and TLS | Name | Key | Description | Default | | ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **TLS** | `tls` | If `true`, enables TLS/SSL. If `false`, disables TLS/SSL. Accepted values: `true`, `false`. | `false` | | **TLS Certificate Validation** | `tls.verify` | If `on`, and if `tls` is `true`, enables TLS/SSL certificate validation. If `off`, disables TLS/SSL certificate validation. Accepted values: `on`, `off`. | `on` | | **TLS Debug Level** | `tls.debug` | Sets TLS debug verbosity level. Accepted values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose). | `1` | | **CA Certificate File Path** | `tls.ca_file` | Absolute path to CA certificate file. | *none* | | **Certificate File Path** | `tls.crt_file` | Absolute path to certificate file. | *none* | | **Private Key File Path** | `tls.key_file` | Absolute path to private key file. | *none* | | **Private Key Path Password** | `tls.key_passwd` | Password for private key file. | *none* | | **TLS SNI Hostname Extension** | `tls.vhost` | Hostname to be used for TLS SNI extension. | *none* | # Test plugin settings Source: https://docs.chronosphere.io/ingest/pipeline/plugins/test-settings Learn how to test plugin settings and ensure data is flowing through your pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. When you [create or modify a pipeline](/ingest/pipeline/v2/build/create-modify), you can test the source and destination plugins in that pipeline to ensure their configuration settings are correct. Chronosphere recommends testing plugins individually rather than as a group, which can help identify and isolate issues with each plugin. ## Test a source plugin To confirm that a source plugin is configured correctly, route data from that source to a basic test destination. 1. Add the source plugin of your choice to a pipeline and configure its settings accordingly. 2. Add the [Standard output](/ingest/pipeline/plugins/destination-plugins/standard-output) destination plugin to the same pipeline. This plugin lets you view telemetry data as it passes through your pipeline in the Telemetry Pipeline web interface. 3. In Telemetry Pipeline, navigate to the [pipeline details](/ingest/pipeline/navigate#pipeline-details-v2-pipelines) page. 4. In the [**Pipeline Output**](/ingest/pipeline/navigate#pipeline-output-v2-pipelines) section, click **Get Latest Logs**. 5. Confirm that **Pipeline Output** displays data from your source. For example, if you're using the [Kubernetes Events](/ingest/pipeline/plugins/source-plugins/kubernetes) source plugin, this output should include data generated by a Kubernetes cluster. If **Pipeline Output** displays data from your source, the associated source plugin is configured correctly. You can either remove the Standard output plugin from your pipeline or copy the source plugin's configuration settings to use them in a different pipeline. ## Test a destination plugin To confirm that a destination plugin is configured correctly, send basic test data to that destination. 1. Add the destination plugin of your choice to a pipeline and configure its settings accordingly. 2. Choose a source plugin to generate test data that's compatible with your destination: * The [Datagen](/ingest/pipeline/plugins/source-plugins/datagen), [Dummy HTTP](/ingest/pipeline/plugins/source-plugins/dummy-http), and [Mock Data](/ingest/pipeline/plugins/source-plugins/mock-data) source plugins generate test logs. * The [Dummy logs, metrics, and traces](/ingest/pipeline/plugins/source-plugins/dummy) source plugin can generate all three telemetry types. 3. Add that source plugin to the same pipeline and configure its settings accordingly. 4. Access the storage location associated with your destination plugin to confirm whether that destination is receiving test data. For example, if you're using the [Amazon S3](/ingest/pipeline/plugins/destination-plugins/amazon-s3) destination plugin, look for test logs in your S3 bucket. If your destination is receiving test data, the associated destination plugin is configured correctly. You can either remove the testing source plugin from your pipeline or copy the destination plugin's configuration settings to use them in a different pipeline. # Processing rules Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules Transform the data that passes through your telemetry pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. ```mermaid actions={false} theme={null} flowchart LR accTitle: Pipeline flow accDescr: A simplified representation of a telemetry pipeline, which includes a source plugin, parser, processing rules, and destination plugin. subgraph Pipeline flow direction LR A(Source plugin) B(Parser) C(Processing rules) D(Destination plugin) A --> B --> C --> D style C stroke-width:4px end ``` Processing rules transform the data that passes through your telemetry pipeline. You can use a variety of rules to perform different operations on data in transit between its source and its destination. Some example use cases for processing rules include: * Adding a new field to each log for easier debugging and troubleshooting. * Redacting sensitive values to preserve user privacy. * Removing unnecessary fields to improve the signal-to-noise ratio of your data. * Converting data from one format to another. * Turning unstructured data into structured data. * Aggregating logs into metrics to reduce data volume while retaining key insights. ## Overview ```mermaid actions={false} theme={null} graph TB accTitle: Processing rules diagram. accDescr: Data transforms as it passes through the logic of a processing rule. raw{"{'event': 'login',
'token': 'hunter2'}"} --> rule[Processing rules] subgraph rule[Rule: redact/mask value] direction LR logic[key: token
regex: ^.+$
replaceChar: *] end rule --> processed{"{'event': 'login',
'token': '*******'}"} style raw stroke:#495057,stroke-width:2px style rule stroke:#495057,stroke-width:2px style processed stroke:#495057,stroke-width:2px style logic stroke-dasharray: 5 5 ``` Each built-in processing rule transforms data in a specific way. You can combine multiple processing rules to create complex transformation logic suited for your telemetry data and storage format. Processing rules are applied at the [plugin](/ingest/pipeline/plugins) level, and each plugin can have multiple associated processing rules. Any rules associated with a [source plugin](/ingest/pipeline/plugins/source-plugins) are applied after the data from that plugin has been parsed by a [file-based parser](/ingest/pipeline/v2/build/parsers), if applicable. Any rules associated with a [destination plugin](/ingest/pipeline/plugins/destination-plugins) are applied before the data leaves your pipeline. ## Format Processing rules are run one at a time, from top to bottom. If you add multiple processing rules to the same pipeline for the same [telemetry type](#telemetry-types), the output from your first rule becomes the input for your second rule, the output from your second rule becomes the input for your third rule, and so on. Most processing rules are compatible with most data formats. Processing rules are also designed to skip any incompatible records or records that can't be processed, which means you can create rules even if they only apply to a subset of your data. For example, you can apply a broad rule to remove a certain field from your logs even if some logs that travel through your pipeline don't contain that field. ### Telemetry types > Requires Core Operator version 3.1.0 or later and pipeline agent version 24.7.3 or later. Processing rules support logs, metrics, and traces. You can create processing rules for multiple telemetry types within the same pipeline, but each processing rule is applied only to its specified telemetry type. For example, if you create a [search/replace value](/ingest/pipeline/processing-rules/search-replace-value) processing rule for metrics, this rule won't affect any logs or traces that pass through your pipeline, even if those logs or traces contain a matching key. When raw log data passes through at least one processing rule, the data receives a new `log` field for each event. This `log` field lets you treat each event as a single unit of data. Structured log data such as JSON doesn't receive a `log` field because you can already break structured data into discrete events. ### Regex engines > Requires Telemetry Pipeline version 2.9.0 or later. Some processing rules, like [block records](/ingest/pipeline/processing-rules/block-records) and [rename keys](/ingest/pipeline/processing-rules/rename-keys), accept regular expressions. For most of these rules, you can specify one of the following engines to parse your rule: * [PCRE2](https://www.pcre.org/current/doc/html/) (default) * [Oniguruma](https://github.com/kkos/oniguruma) * [POSIX](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/regex.h.html) * [GNU](https://www.gnu.org/software/gnulib/manual/html_node/Regular-expressions.html) * [TRE](https://github.com/laurikari/tre/) ### Record accessor syntax If your raw data is in JSON format, you can use record accessor syntax to extract nested fields within a larger JSON object. To extract a nested field inside a standard object, use the following syntax: ```text theme={null} $objectName.fieldName ``` To extract a nested field inside an array, use the following syntax: ```text theme={null} $objectName.arrayName[X] ``` For example, given the following JSON object: ```json theme={null} { "log": "1234", "kubernetes": { "pod_name": "mypod-0", "labels": [ { "app": "myapp", "env": "prod" } ] } } ``` The expression `$kubernetes.pod_name` resolves to `mypod-0`, and the expression `$kubernetes.labels[0]` resolves to `"app": "myapp"`. If the name of a field or parent object contains periods, wrap the name in quotes. For example, to access the field `k8s.pod.uid` inside the object `attributes`, use the expression `$attributes."k8s.pod.uid"`. ## Add processing rules to your pipeline 1. Sign in to the [Telemetry Pipeline web interface](https://core.calyptia.com/). 2. Open the project that contains the pipeline to which you want to add processing rules. 3. Navigate to **Core Instances**, then select the pipeline to which you want to add processing rules. 4. Click **Edit**. 5. Click the node in the middle of the configuration diagram. 6. In the dialog that appears, select an option from the **Telemetry type** tab. 7. Click  **Add New Rule** to open the processing rules menu. 8. Select a processing rule from the available list. 9. Configure the available settings for that processing rule, and then click **Apply**. 10. Optional: Repeat steps 5 through 8 to add additional processing rules. If you add multiple rules, you can drag them to change the order in which they run. 11. Optional: Add test input and then click  **Preview Output** to preview the output of your processing rules. 12. Click **Apply Processing Rules**  to finalize your processing rules, and then click **Save and Deploy**  to save your pipeline settings. Use the toggle next to a processing rule to enable or disable that rule. ## Use pipeline logs as preview input data You can use the **Input** and **Output** panes in the processing rules builder to preview the effects of your processing rules. When you edit processing rules for most pipelines, or when you use the processing rules playground, the **Input** pane displays sample logs by default. However, when you edit certain eligible pipelines, the **Input** pane automatically fetches the latest logs from that pipeline. You can also manually fetch logs from eligible pipelines to use as preview input data in the processing rules playground or when editing any other pipelines. To be an eligible preview input source, a pipeline must be active and must include the [Standard output](/ingest/pipeline/plugins/destination-plugins/standard-output) destination plugin. This plugin is necessary because Chronosphere can't access the telemetry data that passes through your pipelines without it. To use a pipeline's logs as preview input data, choose an eligible pipeline from the **Select pipeline** drop-down menu, then click **Get Latest Logs**. This takes a snapshot of the logs currently passing through your pipeline and displays that snapshot in the **Input** pane. If you click  **Preview Output**, the **Output** pane will show how the logs in that snapshot would change based on the processing rules that you've configured. However, processing rules won't actually modify the previewed logs unless you save and apply changes to the pipeline from which those logs originate. To use sample logs as preview input, open the **Select pipeline** drop-down menu and click **Sample logs**. You can also overwrite the logs in the **Input** pane with any text of your choosing. ## Save and load templates To create a template from your current processing rule settings, click  **Export** to save a JSON template file to your device. You can also click  **Save** to save your template within the Telemetry Pipeline web interface, which makes that template available to any other users in your current project. To load an existing template, click  **Import** and select a JSON template file stored on your device, or use the **Select Template** drop-down menu to choose one of the templates available in your current project. Chronosphere also offers several processing rule templates in the [`processing-templates` GitHub repository](https://github.com/chronosphereio/processing-templates). ## Available processing rules Telemetry Pipeline offers the following processing rules: | Processing rule | Description | | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [**Set key/value**](/ingest/pipeline/processing-rules/add-set-key-value) | Adds the same key-value pair to every record. | | [**Aggregate records**](/ingest/pipeline/processing-rules/aggregate-records) | Transforms incoming logs into computed metrics at periodic intervals. | | [**Allow keys**](/ingest/pipeline/processing-rules/allow-keys) | Preserves any keys that match a specified regular expression, and removes all other keys. | | [**Allow records**](/ingest/pipeline/processing-rules/allow-records) | Preserves any records that contain a key whose value matches a specified regular expression, and removes all other records. | | [**Block keys**](/ingest/pipeline/processing-rules/block-keys) | Removes keys that match a regular expression. | | [**Block records**](/ingest/pipeline/processing-rules/block-records) | Removes records that contain a key whose value matches a specified regular expression. | | [**Copy keys**](/ingest/pipeline/processing-rules/copy-keys) | Copies the value of a specified source key to the value of a specified destination key. | | [**Custom Lua**](/ingest/pipeline/processing-rules/custom-lua) | Write custom Lua scripts to transform your telemetry data. | | [**Decode CSV**](/ingest/pipeline/processing-rules/decode-csv) | Transforms log data from CSV format to JSON. | | [**Decode JSON**](/ingest/pipeline/processing-rules/decode-json) | Transforms an escaped JSON string into a structured JSON object. | | [**Deduplicate records**](/ingest/pipeline/processing-rules/deduplicate-records) | Searches for any records that contain identical key-value data within a specified time frame, then removes all but the earliest of those records. | | [**Delete key**](/ingest/pipeline/processing-rules/delete-key) | Deletes a specified key and its associated value from all records. | | [**Encode CSV**](/ingest/pipeline/processing-rules/encode-csv) | Transforms log data from JSON to CSV format. | | [**Encode JSON**](/ingest/pipeline/processing-rules/encode-json) | Transforms a JSON object into an escaped string. | | [**Extract keys/values**](/ingest/pipeline/processing-rules/extract-keys-values) | Uses a regular expression to search for key-value pairs inside a string, then creates a structured object to store those key-value pairs. | | [**Flatten subrecord**](/ingest/pipeline/processing-rules/flatten-subrecord) | Uses regular expression to search for key-value pairs inside a JSON object, then either moves or copies any applicable key-value pairs to the top level of the record. | | [**Hash key**](/ingest/pipeline/processing-rules/hash-key) | Copies the value of a specified source key, hashes that value, then stores the hashed copy in a specified destination key. | | [**Join records**](/ingest/pipeline/processing-rules/join-records) | Combined values from multiple records into an array of values within a single record. | | [**Lift submap**](/ingest/pipeline/processing-rules/lift-submap) | Uses regular expressions to search for key-value pairs inside a JSON object, then either moves or copies any applicable key-value pairs out of the JSON object and into a higher level of the record. | | [**Multiline join**](/ingest/pipeline/processing-rules/multiline-join) | Combines multiple logs into a single log by looking for repeating patterns in log data. | | [**Nest keys**](/ingest/pipeline/processing-rules/nest-keys) | Moves the value of a specified source key into an object nested under a specified destination key. | | [**Parse**](/ingest/pipeline/processing-rules/parse) | Uses a regular expression to search for values inside a string and assign a key to each value, then stores those key-value pairs in a structured object. | | [**Parse number**](/ingest/pipeline/processing-rules/parse-number) | Uses a regular expression to transform a key's value from a string to a number. | | [**Random sampling**](/ingest/pipeline/processing-rules/random-sampling) | Preserves a percentage of records that pass through your pipeline and discards the rest. | | [**Redact/mask value**](/ingest/pipeline/processing-rules/redact-mask-value) | Obscures all or part of a specified key's value by replacing the original string with a series of repeated characters. | | [**Rename keys**](/ingest/pipeline/processing-rules/rename-keys) | Changes the name of a specified key. | | [**Search/replace value**](/ingest/pipeline/processing-rules/search-replace-value) | Uses regular expressions to search for a value inside a string, then replaces that value with a different specified value. | | [**Split record**](/ingest/pipeline/processing-rules/split-record) | Splits an array of JSON objects into a series of standalone records. | # Set key/value Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/add-set-key-value Learn about the set key/value processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The set key/value [processing rule](/ingest/pipeline/processing-rules) adds the same key-value pair to every record that passes through your pipeline. If a record contains a matching key, this rule overwrites that key's associated value with the value you specify. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ----------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Key** | `key` | Required. The key to add to each record. You can use [record accessor syntax](/ingest/pipeline/processing-rules#record-accessor-syntax) to add or overwrite keys nested inside a parent object. | *none* | | **Value** | `value` | Required. The value to associate with your key. This value is identical for every instance of **Key**. | *none* | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Example Using the set key/value rule lets you add identifying information to events, such as including the source from which they originate or their associated user action. For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase", "source": "storefront"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase", "source": "storefront"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase", "source": "storefront"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase", "source": "storefront"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase", "source": "storefront"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"} ``` A rule with the **Key** value `source` and the **Value** value `acme-wholesale` returns the following result: ```json theme={null} {"source":"acme-wholesale","user_id":3,"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:41.64283645Z"} {"source":"acme-wholesale","user_id":4,"page_id":10,"action":"purchase","timestamp":"2023-03-28T09:08:42.643343109Z"} {"source":"acme-wholesale","user_id":1,"page_id":50,"action":"click","timestamp":"2023-03-28T09:08:48.643600498Z"} {"source":"acme-wholesale","user_id":5,"page_id":40,"action":"purchase","timestamp":"2023-03-28T09:08:50.643773688Z"} {"source":"acme-wholesale","user_id":1,"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:51.643932272Z"} {"source":"acme-wholesale","user_id":2,"page_id":40,"action":"click","timestamp":"2023-03-28T09:08:56.644080944Z"} {"source":"acme-wholesale","user_id":3,"page_id":30,"action":"click","timestamp":"2023-03-28T09:09:03.64425954Z"} {"source":"acme-wholesale","user_id":1,"page_id":20,"action":"view","timestamp":"2023-03-28T09:09:03.644317046Z"} {"source":"acme-wholesale","user_id":2,"page_id":50,"action":"purchase","timestamp":"2023-03-28T09:09:10.64447719Z"} {"source":"acme-wholesale","user_id":2,"page_id":10,"action":"view","timestamp":"2023-03-28T09:09:17.644810963Z"} {"source":"acme-wholesale","user_id":1,"page_id":50,"action":"view","timestamp":"2023-03-28T09:09:20.644994805Z"} ``` A `source` key is now included for each record, with the value `acme-wholesale`. Records that already had a key named `source` had their values replaced with the new value. # Aggregate records Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/aggregate-records Learn about the aggregate records processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The aggregate records [processing rule](/ingest/pipeline/processing-rules) transforms incoming logs into computed metrics at periodic intervals, and then deletes the original logs. When the aggregate records rule waits for data to accumulate during the specified time window, the pipeline [buffers](/ingest/pipeline/v2/configure/backpressure) that data. Increasing the value of the **Time window** parameter also increases the memory load on your pipeline. For example, if 100,000 records pass through your pipeline during the specified time period, and those records are 1 kB each, the aggregate records rule will add approximately 100 MB of memory load. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ---------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | | **Time window** | `window` | Required. How long to wait (in seconds) for data to accumulate in your pipeline before computing metrics from that data. For each interval that elapses, a new set of metrics is computed. | *none* | | **Select keys** | `keys` | Required. Any logs that contain matching values for all of the specified keys will be grouped together during compute operations. This parameter must be formatted as a JSON array of strings, like `["keyName1","keyName2"]`. | *none* | | **Compute keys** | `compute` | Required. The keys and computed metrics to include in your output data. This parameter must be formatted as a JSON object. For more information, see [Compute syntax](#compute-syntax). | *none* | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Compute syntax The aggregate records processing rule offers three compute functions: `sum`, `average`, and `count`. To use these functions, the **Compute keys** configuration parameter expects a JSON object with the following syntax: ```json theme={null} {"newKey1":["count"],"newKey2":["average","KEY_TO_COMPUTE"],"newKey3":["sum","KEY_TO_COMPUTE"]} ``` The resulting computed metrics are formatted as a JSON object with the following characteristics: ```json theme={null} { "selectKey": "value", "newKey1": "countMetric", "newKey2": "averageMetric", "newKey3": "sumMetric", } ``` * `"selectKey": "value"`: This key and value match the **Select keys** parameter, and together represent the log groups for which metrics are computed. If **Select keys** includes multiple keys, all keys and their corresponding values are included. * `countMetric`: The number of times that `"selectKey": "value"` occurred within the specified time window. * `averageMetric`: The mean value of *`KEY_TO_COMPUTE`* within the specified time window and group. * `sumMetric`: The sum of all *`KEY_TO_COMPUTE`* values within the specified time window and group. ## Example Using the aggregate records rule lets you retain information about the logs that pass through your pipeline without needing to retain each individual log. For example, given this sample JSON data: ```json theme={null} {"name": "Sophia", "profession": "designer", "age": 29,"projects": 2} {"name": "William", "profession": "programmer", "age": 45,"projects": 0} {"name": "Mia", "profession": "chef", "age": 32,"projects": 5} {"name": "Benjamin", "profession": "architect", "age": 51,"projects": 1} {"name": "Ava", "profession": "designer", "age": 27,"projects": 1} {"name": "Michael", "profession": "programmer", "age": 38,"projects": 3} {"name": "Abigail", "profession": "designer", "age": 42,"projects": 0} {"name": "Daniel", "profession": "architect", "age": 35,"projects": 4} {"name": "Emma", "profession": "programmer", "age": 48,"projects": 1} {"name": "Jacob", "profession": "chef", "age": 31,"projects": 2} {"name": "Olivia", "profession": "designer", "age": 24,"projects": 2} {"name": "Matthew", "profession": "chef", "age": 39,"projects": 0} {"name": "Isabella", "profession": "programmer", "age": 28,"projects": 3} {"name": "Ethan", "profession": "architect", "age": 46,"projects": 1} {"name": "Avery", "profession": "designer", "age": 33,"projects": 1} ``` A processing rule with the **Time window** value `60`, the **Select keys** value `["profession"]`, and the **Compute keys** value `{"headcount":["count"],"averageAge":["average","age"],"totalProjects":["sum","projects"]}` returns the following result: ```json theme={null} {"headcount":5,"averageAge":31,"totalProjects":6,"profession":"designer"} {"headcount":4,"averageAge":39.75,"totalProjects":7,"profession":"programmer"} {"headcount":3,"averageAge":34,"totalProjects":7,"profession":"chef"} {"headcount":3,"averageAge":44,"totalProjects":6,"profession":"architect"} ``` Because the original data set's `profession` key had four possible values, this rule created four groups (`designer`, `programmer`, `chef`, and `architect`), and then computed the following metrics for each group: * `averageAge`: The mean `age` value of every person within that group. * `headcount`: The number of people within that group. * `totalProjects`: The combined number of projects completed by the people within that group. After computing these metrics, this rule deleted the original logs. # Allow keys Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/allow-keys Learn about the allow keys processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The allow keys [processing rule](/ingest/pipeline/processing-rules) preserves any keys that match a specified regular expression, and then removes all other keys in each record. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Regex** | `regex` | Required. The regular expression that determines which keys to preserve. | *none* | | **Match case** checkbox | `matchCase` | Indicates whether the regular expression is case-sensitive. | Not selected | | **Regex engine** | `regexEngine` | Required. The [engine](/ingest/pipeline/processing-rules#regex-engines) to parse your regular expression. Accepted values: `GNU`, `Oniguruma`, `PCRE2`, `POSIX`, `TRE`. | `PCRE2` | | **Nested access pattern** | `nestedPath` | The key of the highest object level to evaluate. If specified, the allow keys rule ignores any keys not contained within the specified object. If unspecified, this rule evaluates all top-level keys, but no keys nested within top-level keys. You can also use [record accessor syntax](/ingest/pipeline/processing-rules#record-accessor-syntax) to reference keys nested within another nested object. | *none* | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Examples Using the allow keys rule lets you pare down telemetry data by retaining only the data you explicitly choose to keep. ### Flat example For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"} ``` A processing rule with the **Regex** value `user` returns the following result: ```json theme={null} {"user_id":3} {"user_id":4} {"user_id":1} {"user_id":5} {"user_id":1} {"user_id":2} {"user_id":3} {"user_id":1} {"user_id":2} {"user_id":2} {"user_id":1} ``` ### Nested example You can also use the allow keys rule to selectively retain information within a nested object. For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"vip":"no","id":3},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"vip":"yes","id":4},"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"vip":"no","id":1},"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"vip":"yes","id":5},"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"vip":"no","id":1},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"vip":"yes","id":2},"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"vip":"no","id":3},"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"vip":"no","id":1},"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"vip":"yes","id":2},"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"vip":"yes","id":2},"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"vip":"no","id":1},"page_id":50,"action":"view"} ``` A processing rule with the **Regex** value `vip` and the **Nested access pattern** value `user` returns the following result: ```json theme={null} {"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:41.64283645Z","user":{"vip":"no"}} {"page_id":10,"action":"purchase","timestamp":"2023-03-28T09:08:42.643343109Z","user":{"vip":"yes"}} {"page_id":50,"action":"click","timestamp":"2023-03-28T09:08:48.643600498Z","user":{"vip":"no"}} {"page_id":40,"action":"purchase","timestamp":"2023-03-28T09:08:50.643773688Z","user":{"vip":"yes"}} {"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:51.643932272Z","user":{"vip":"no"}} {"page_id":40,"action":"click","timestamp":"2023-03-28T09:08:56.644080944Z","user":{"vip":"yes"}} {"page_id":30,"action":"click","timestamp":"2023-03-28T09:09:03.64425954Z","user":{"vip":"no"}} {"page_id":20,"action":"view","timestamp":"2023-03-28T09:09:03.644317046Z","user":{"vip":"no"}} {"page_id":50,"action":"purchase","timestamp":"2023-03-28T09:09:10.64447719Z","user":{"vip":"yes"}} {"page_id":10,"action":"view","timestamp":"2023-03-28T09:09:17.644810963Z","user":{"vip":"yes"}} {"page_id":50,"action":"view","timestamp":"2023-03-28T09:09:20.644994805Z","user":{"vip":"no"}} ``` This rule retained the `vip` field within the `user` object and removed all other fields within `user`. However, because the processing rule's scope was limited to `user`, the rule didn't affect the `timestamp`, `page_id`, or `action` fields. ## Related rules For a processing rule with the opposite effect, see [block keys](/ingest/pipeline/processing-rules/block-keys). # Allow records Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/allow-records Learn about the allow records processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The allow records [processing rule](/ingest/pipeline/processing-rules) preserves any records that contain a key whose value matches a specified regular expression, and then removes all other records. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ----------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Key** | `key` | Required. The key whose value to validate against. If this key isn't present in a given record, that record is removed. | `log` | | **Regex** | `regex` | Required. The regular expression that determines which records to preserve. This expression is evaluated against the value of the specified key. | *none* | | **Match case** checkbox | `matchCase` | Indicates whether the regular expression is case-sensitive. | Not selected | | **Regex engine** | `regexEngine` | Required. The [engine](/ingest/pipeline/processing-rules#regex-engines) to parse your regular expression. Accepted values: `GNU`, `Oniguruma`, `PCRE2`, `POSIX`, `TRE`. | `PCRE2` | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Example Using the allow records rule lets you pare down telemetry data by retaining only the records with key-value pairs that you explicitly choose to keep. For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"} ``` A processing rule with the **Key** value `action` and the **Regex** value `purchase` returns the following result: ```json theme={null} {"user_id":3,"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:41.64283645Z"} {"user_id":4,"page_id":10,"action":"purchase","timestamp":"2023-03-28T09:08:42.643343109Z"} {"user_id":5,"page_id":40,"action":"purchase","timestamp":"2023-03-28T09:08:50.643773688Z"} {"user_id":1,"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:51.643932272Z"} {"user_id":2,"page_id":50,"action":"purchase","timestamp":"2023-03-28T09:09:10.64447719Z"} ``` This rule retained the records associated with user purchases and removed the records associated with other actions, such as pageviews or clicks. ## Related rules For a processing rule with the opposite effect, see [block records](/ingest/pipeline/processing-rules/block-records). # Block keys Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/block-keys Learn about the block keys processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The block keys [processing rule](/ingest/pipeline/processing-rules) removes any keys that match a specified regular expression, and then preserves all other keys in each record. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Regex** | `regex` | Required. The regular expression that determines which keys to remove. | *none* | | **Match case** checkbox | `matchCase` | Indicates whether the regular expression is case-sensitive. | Not selected | | **Regex engine** | `regexEngine` | Required. The [engine](/ingest/pipeline/processing-rules#regex-engines) to parse your regular expression. Accepted values: `GNU`, `Oniguruma`, `PCRE2`, `POSIX`, `TRE`. | `PCRE2` | | **Nested access pattern** | `nestedPath` | The key of the highest object level to evaluate. If specified, the block keys rule ignores any keys not contained within the specified object. If unspecified, this rule evaluates all top-level keys, but no keys nested within top-level keys. You can also use [record accessor syntax](/ingest/pipeline/processing-rules#record-accessor-syntax) to reference keys nested within another nested object. | *none* | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Examples Using the block keys rule lets you pare down telemetry data by eliminating data you don't want to keep. ### Flat example For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"} ``` A processing rule with the **Regex** value `page_id` returns the following result: ```json theme={null} {"action":"purchase","user_id":3,"timestamp":"2023-03-28T09:08:41.64283645Z"} {"action":"purchase","user_id":4,"timestamp":"2023-03-28T09:08:42.643343109Z"} {"action":"click","user_id":1,"timestamp":"2023-03-28T09:08:48.643600498Z"} {"action":"purchase","user_id":5,"timestamp":"2023-03-28T09:08:50.643773688Z"} {"action":"purchase","user_id":1,"timestamp":"2023-03-28T09:08:51.643932272Z"} {"action":"click","user_id":2,"timestamp":"2023-03-28T09:08:56.644080944Z"} {"action":"click","user_id":3,"timestamp":"2023-03-28T09:09:03.64425954Z"} {"action":"view","user_id":1,"timestamp":"2023-03-28T09:09:03.644317046Z"} {"action":"purchase","user_id":2,"timestamp":"2023-03-28T09:09:10.64447719Z"} {"action":"view","user_id":2,"timestamp":"2023-03-28T09:09:17.644810963Z"} {"action":"view","user_id":1,"timestamp":"2023-03-28T09:09:20.644994805Z"} ``` This rule removed the `page_id` key from each log and retained all other keys. ### Nested example You can also use the allow keys rule to selectively remove information within a nested object. For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"vip":"no","id":3},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"vip":"yes","id":4},"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"vip":"no","id":1},"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"vip":"yes","id":5},"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"vip":"no","id":1},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"vip":"yes","id":2},"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"vip":"no","id":3},"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"vip":"no","id":1},"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"vip":"yes","id":2},"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"vip":"yes","id":2},"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"vip":"no","id":1},"page_id":50,"action":"view"} ``` A processing rule with the **Regex** value `vip` and the **Nested access pattern** value `user` returns the following result: ```json theme={null} {"user":{"id":3},"action":"purchase","page_id":30,"timestamp":"2023-03-28T09:08:41.64283645Z"} {"user":{"id":4},"action":"purchase","page_id":10,"timestamp":"2023-03-28T09:08:42.643343109Z"} {"user":{"id":1},"action":"click","page_id":50,"timestamp":"2023-03-28T09:08:48.643600498Z"} {"user":{"id":5},"action":"purchase","page_id":40,"timestamp":"2023-03-28T09:08:50.643773688Z"} {"user":{"id":1},"action":"purchase","page_id":30,"timestamp":"2023-03-28T09:08:51.643932272Z"} {"user":{"id":2},"action":"click","page_id":40,"timestamp":"2023-03-28T09:08:56.644080944Z"} {"user":{"id":3},"action":"click","page_id":30,"timestamp":"2023-03-28T09:09:03.64425954Z"} {"user":{"id":1},"action":"view","page_id":20,"timestamp":"2023-03-28T09:09:03.644317046Z"} {"user":{"id":2},"action":"purchase","page_id":50,"timestamp":"2023-03-28T09:09:10.64447719Z"} {"user":{"id":2},"action":"view","page_id":10,"timestamp":"2023-03-28T09:09:17.644810963Z"} {"user":{"id":1},"action":"view","page_id":50,"timestamp":"2023-03-28T09:09:20.644994805Z"} ``` This rule removed the `vip` key within the `user` object and retained all other keys within `user`. However, because the processing rule's scope was limited to `user`, the rule didn't affect the `timestamp`, `page_id`, or `action` fields. ## Related rules For a processing rule with the opposite effect, see [allow keys](/ingest/pipeline/processing-rules/allow-keys). # Block records Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/block-records Learn about the block records processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The block records [processing rule](/ingest/pipeline/processing-rules) removes any records that contain a key whose value matches a specified regular expression, and then preserves all other records. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ----------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Key** | `key` | Required. The key whose value to validate against. If this key isn't present in a given record, that record is preserved. | `log` | | **Regex** | `regex` | Required. The regular expression that determines which records to remove. This expression is evaluated against the value of the specified key. | *none* | | **Match case** checkbox | `matchCase` | Indicates whether the regular expression is case-sensitive. | Not selected | | **Regex engine** | `regexEngine` | Required. The [engine](/ingest/pipeline/processing-rules#regex-engines) to parse your regular expression. Accepted values: `GNU`, `Oniguruma`, `PCRE2`, `POSIX`, `TRE`. | `PCRE2` | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Example Using the block records rule lets you pare down telemetry data by eliminating records with data you don't want to keep. For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"} ``` A processing rule with the **Key** value `action` and the **Regex** value `purchase` returns the following result: ```json theme={null} {"user_id":1,"page_id":50,"action":"click","timestamp":"2023-03-28T09:08:48.643600498Z"} {"user_id":2,"page_id":40,"action":"click","timestamp":"2023-03-28T09:08:56.644080944Z"} {"user_id":3,"page_id":30,"action":"click","timestamp":"2023-03-28T09:09:03.64425954Z"} {"user_id":1,"page_id":20,"action":"view","timestamp":"2023-03-28T09:09:03.644317046Z"} {"user_id":2,"page_id":10,"action":"view","timestamp":"2023-03-28T09:09:17.644810963Z"} {"user_id":1,"page_id":50,"action":"view","timestamp":"2023-03-28T09:09:20.644994805Z"} ``` This rule removed the records associated with user purchases and preserved the records associated with other actions, such as pageviews or clicks. ## Related rules For a processing rule with the opposite effect, see [allow records](/ingest/pipeline/processing-rules/allow-records). # Copy keys Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/copy-keys Learn about the copy keys processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The copy keys [processing rule](/ingest/pipeline/processing-rules) copies the value of a specified source key to the value of a specified destination key. This destination can either be a new or existing key. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ | | **Source key** | `src` | Required. The key whose value to copy. | *none* | | **Destination key** | `dst` | Required. The key whose value is set to match the source key. If no matching key exists, this rule creates a new key and then set its value accordingly. | *none* | | **Regex engine** | `regexEngine` | Required. The [engine](/ingest/pipeline/processing-rules#regex-engines) to parse your regular expression. Accepted values: `GNU`, `Oniguruma`, `PCRE2`, `POSIX`, `TRE`. | `PCRE2` | | **Regex mode** checkbox | `regexMode` | Indicates whether the values of the source key and destination key are regular expressions. If selected, **Source key** is treated as a regular expression and **Destination key** is treated as a substitution pattern. | Not selected | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Example Using the block keys rule lets you copy the same value across multiple keys or create a new key with the same value as an existing key. For example, given the following sample website log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"} ``` A processing rule with the **Source key** value `page_id` and the **Destination key** value `product_id` returns the following result: ```json theme={null} {"user_id":3,"product_id":30,"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:41.64283645Z"} {"user_id":4,"product_id":10,"page_id":10,"action":"purchase","timestamp":"2023-03-28T09:08:42.643343109Z"} {"user_id":1,"product_id":50,"page_id":50,"action":"click","timestamp":"2023-03-28T09:08:48.643600498Z"} {"user_id":5,"product_id":40,"page_id":40,"action":"purchase","timestamp":"2023-03-28T09:08:50.643773688Z"} {"user_id":1,"product_id":30,"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:51.643932272Z"} {"user_id":2,"product_id":40,"page_id":40,"action":"click","timestamp":"2023-03-28T09:08:56.644080944Z"} {"user_id":3,"product_id":30,"page_id":30,"action":"click","timestamp":"2023-03-28T09:09:03.64425954Z"} {"user_id":1,"product_id":20,"page_id":20,"action":"view","timestamp":"2023-03-28T09:09:03.644317046Z"} {"user_id":2,"product_id":50,"page_id":50,"action":"purchase","timestamp":"2023-03-28T09:09:10.64447719Z"} {"user_id":2,"product_id":10,"page_id":10,"action":"view","timestamp":"2023-03-28T09:09:17.644810963Z"} {"user_id":1,"product_id":50,"page_id":50,"action":"view","timestamp":"2023-03-28T09:09:20.644994805Z"} ``` This rule copied the value of each log's `page_id` key to the value of a new key called `product_id`. # Custom Lua Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/custom-lua Learn about the custom Lua processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. You can use the custom Lua [processing rule](/ingest/pipeline/processing-rules) to write your own Lua scripts for transforming telemetry data. This rule uses [LuaJIT 5.1](https://www.lua.org/manual/5.1/manual.html#5) and runs in protected mode to ensure that malformed Lua code doesn't harm pipelines. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ----------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Script** | `script` | Required. Your custom Lua script. | *none* | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. You can also use [GPT generation](#gpt-generation) to treat this comment as the prompt for a new Lua script. | *none* | ## GPT generation Keep in mind that this feature is experimental, and that Chronosphere can't guarantee the accuracy or quality of generated scripts. To enable or disable GPT generation, see [Project settings](/ingest/pipeline/administer/project-settings#enable-or-disable-gpt-generation). The custom Lua processing rule lets you turn text descriptions into scripts by using generative pre-trained transformer (GPT) generation. To use this feature, enter a description of your script in the **Comment** text box, and then click **Generate Lua Script From Comment With Gpt Experimental**. ## Best practices To ensure that custom Lua scripts run efficiently within your pipelines, Chronosphere recommends following these best practices: * Test new scripts in the processing rules playground. Before adding a custom Lua script to an active pipeline, try running the script in the processing rules playground in the Telemetry Pipeline UI. This lets you test your script on sample input data, view the script's transformed output, and catch syntax errors. * Add `print` and `error` statements to your scripts. These statements help you track the value of variables and see details about specific errors when they occur. * Use profiling. In the processing rules playground, you can toggle the **Enable profiling** setting to view more information about a script's transformed output, including how long it took for that script to run. ## Example scripts The following examples show several possible types of Lua scripts you can incorporate into your pipeline. ### Conditional statement You can use conditional statements to transform data that meets certain criteria. This example script looks for a key named `log`, and then determines whether its value is `line`. If both of these conditions are true, the script sets the value of `log` to `Y`. ```lua theme={null} return function(tag, ts, record) if record['log'] ~= nil and record['log'] == 'line' then record['log'] = 'Y' end return 1, ts, record end ``` ### Reference to a secret If your pipeline includes [secrets](/ingest/pipeline/v2/configure/secrets), you can reference these secrets through Lua scripts. Because secrets are stored as environment variables within a pipeline, this example accesses that environment variable, and then stores its value in a new variable named `secret`. Additionally, this script calls the `resolve` function only if the `secret` key hasn't been defined before, which helps improve performance. ```lua theme={null} function resolve(SECRET_) local token = os.getenv(SECRET_) if token == nil then error(string.format("Could not resolve token")) else return token end end local secret = nil return function(tag, ts, record, code) if secret == nil then secret = resolve("SECRET_") record.new_field = secret else record.new_field = secret end return code, ts, record end ``` ### Call to a third-party API You can use Lua scripts to call a third-party HTTP API, and then use data from that API to perform various actions. This example imports the Lua JSON library to support the JSON response returned by a specific API, then uses a separate function to call that API, and then creates a new key to store the value of a successful API response. ```lua theme={null} local json = require 'json' local function try_curl(url, max_retries, timeout) local retries = 0 while retries < max_retries do local success, result = pcall(function() local command = string.format('curl -s -L --max-time %d %s', timeout, url) local proc = io.popen(command) local output = proc:read('*a') proc:close() return output end) if success and result ~= "" then return result end retries = retries + 1 if retries < max_retries then os.execute("sleep 1") -- Wait for 1 second before retrying end end return nil end return function(tag, ts, record, code) if record.___dummyFlush__ then return 1, ts, record end local url = 'https://gist.githubusercontent.com/agup006/7fbc70a9a821bb39977504afb9ec290e/raw/9ced93796b9eb5a8d0b0d69bdeed630d5c7f2744/test.json' local max_retries = 3 local timeout = 10 -- seconds local response = try_curl(url, max_retries, timeout) if response then local success, result = pcall(json.decode, response) if success then record.json = result else record.json = { message = "failed to parse JSON" } end else record.json = { message = "failed to connect" } end return code, ts, record end ``` ### Reading a pipeline file You can use pipeline files to store arbitrary data, including JSON, and use a Lua script to access that data. This example imports the Lua JSON library, then uploads a file called `schema`, and then converts and returns the data stored in that file. The processing rules playground can't access pipeline files. To test a Lua script that reads files, Chronosphere recommends creating a small pipeline that uses mock data. ```lua theme={null} local json = require 'json' local file_path = "/config/schema" local file = io.open(file_path, "r") if not file then error("Could not open file at " .. file_path) end local jsonschema_content = file:read("*all") file:close() local jsonschema = json.decode(jsonschema_content) if not jsonschema then error("Failed to parse JSON from file at " .. file_path) end return function(tag, ts, record, code) if record.___dummyFlush__ then return 1, ts, record end return code, ts, jsonschema end ``` # Decode CSV Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/decode-csv Learn about the decode CSV processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The decode CSV [processing rule](/ingest/pipeline/processing-rules) transforms log data from CSV format to JSON. Depending on whether you specify header information, the resulting JSON object includes either a set of key-value pairs or an array that splits comma-separated values into individual strings. Additionally, the resulting output always includes a string of the original CSV log data. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | **Source key** | `src` | Required. The key to store your original CSV log entry. Don't change this setting from its default value. | `log` | | **Destination key** | `dst` | Required. The key to store your transformed JSON data. Don't change this setting from its default value. | `decoded_csv` | | **Parse header** checkbox | `parseHeader` | Indicates whether the first entry in your CSV data contains headers. If selected, the decode CSV rule uses these headers as keys to store the values of subsequent rows of data. | Not selected | | **Preset header** | `header` | A list of headers to use as keys for storing the values of each row's data. Use a comma to separate the names of your headers, but don't use spaces between each name. This list must contain the same number of headers as there are columns of data. For example, if your CSV logs contain seven columns of data, you must specify seven header names to use as keys. | *none* | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | If you enable both **Parse header** and **Preset header**, the **Parse header** setting takes precedence. For best results, enable one setting at a time rather than using both simultaneously. ## Examples Using the decode CSV rule lets you turn CSV-stored log data into a more traditional log format for storage and analysis. ### Without headers If you don't specify header information, this rule transforms CSV data into an array. For example, given this sample CSV data: ```text theme={null} 2023-03-28T09:08:41.64283645Z,3,30,purchase 2023-03-28T09:08:42.643343109Z,4,10,purchase 2023-03-28T09:08:48.643600498Z,1,50,click 2023-03-28T09:08:50.643773688Z,5,40,purchase 2023-03-28T09:08:51.643932272Z,1,30,purchase 2023-03-28T09:08:56.644080944Z,2,40,click 2023-03-28T09:09:03.64425954Z,3,30,click 2023-03-28T09:09:03.644317046Z,1,20,view 2023-03-28T09:09:10.64447719Z,2,50,purchase 2023-03-28T09:09:17.644810963Z,2,10,view 2023-03-28T09:09:20.644994805Z,1,50,view ``` A processing rule with the **Source key** value `log` and the **Destination key** value `decoded_csv` returns the following result: ```json theme={null} {"log":"2023-03-28T09:08:41.64283645Z,3,30,purchase","decoded_csv":["2023-03-28T09:08:41.64283645Z","3","30","purchase"]} {"log":"2023-03-28T09:08:42.643343109Z,4,10,purchase","decoded_csv":["2023-03-28T09:08:42.643343109Z","4","10","purchase"]} {"log":"2023-03-28T09:08:48.643600498Z,1,50,click","decoded_csv":["2023-03-28T09:08:48.643600498Z","1","50","click"]} {"log":"2023-03-28T09:08:50.643773688Z,5,40,purchase","decoded_csv":["2023-03-28T09:08:50.643773688Z","5","40","purchase"]} {"log":"2023-03-28T09:08:51.643932272Z,1,30,purchase","decoded_csv":["2023-03-28T09:08:51.643932272Z","1","30","purchase"]} {"log":"2023-03-28T09:08:56.644080944Z,2,40,click","decoded_csv":["2023-03-28T09:08:56.644080944Z","2","40","click"]} {"log":"2023-03-28T09:09:03.64425954Z,3,30,click","decoded_csv":["2023-03-28T09:09:03.64425954Z","3","30","click"]} {"log":"2023-03-28T09:09:03.644317046Z,1,20,view","decoded_csv":["2023-03-28T09:09:03.644317046Z","1","20","view"]} {"log":"2023-03-28T09:09:10.64447719Z,2,50,purchase","decoded_csv":["2023-03-28T09:09:10.64447719Z","2","50","purchase"]} {"log":"2023-03-28T09:09:17.644810963Z,2,10,view","decoded_csv":["2023-03-28T09:09:17.644810963Z","2","10","view"]} {"log":"2023-03-28T09:09:20.644994805Z,1,50,view","decoded_csv":["2023-03-28T09:09:20.644994805Z","1","50","view"]} ``` This rule copied each row of CSV data to the `log` key and used the comma-separated row values to create the `decoded_csv` array. ### With preset headers If you specify header information in the **Preset header** setting, CSV data becomes a series of key-value pairs. For example, given this sample CSV data: ```text theme={null} 2023-03-28T09:08:41.64283645Z,3,30,purchase 2023-03-28T09:08:42.643343109Z,4,10,purchase 2023-03-28T09:08:48.643600498Z,1,50,click 2023-03-28T09:08:50.643773688Z,5,40,purchase 2023-03-28T09:08:51.643932272Z,1,30,purchase 2023-03-28T09:08:56.644080944Z,2,40,click 2023-03-28T09:09:03.64425954Z,3,30,click 2023-03-28T09:09:03.644317046Z,1,20,view 2023-03-28T09:09:10.64447719Z,2,50,purchase 2023-03-28T09:09:17.644810963Z,2,10,view 2023-03-28T09:09:20.644994805Z,1,50,view ``` A processing rule with the **Source key** value `log`, the **Destination key** value `decoded_csv`, and the **Preset header** value `time,user,page,result` returns the following result: ```json theme={null} {"log":"2023-03-28T09:08:41.64283645Z,3,30,purchase","decoded_csv":{"result":"purchase","user":"3","time":"2023-03-28T09:08:41.64283645Z","page":"30"}} {"log":"2023-03-28T09:08:42.643343109Z,4,10,purchase","decoded_csv":{"result":"purchase","user":"4","time":"2023-03-28T09:08:42.643343109Z","page":"10"}} {"log":"2023-03-28T09:08:48.643600498Z,1,50,click","decoded_csv":{"result":"click","user":"1","time":"2023-03-28T09:08:48.643600498Z","page":"50"}} {"log":"2023-03-28T09:08:50.643773688Z,5,40,purchase","decoded_csv":{"result":"purchase","user":"5","time":"2023-03-28T09:08:50.643773688Z","page":"40"}} {"log":"2023-03-28T09:08:51.643932272Z,1,30,purchase","decoded_csv":{"result":"purchase","user":"1","time":"2023-03-28T09:08:51.643932272Z","page":"30"}} {"log":"2023-03-28T09:08:56.644080944Z,2,40,click","decoded_csv":{"result":"click","user":"2","time":"2023-03-28T09:08:56.644080944Z","page":"40"}} {"log":"2023-03-28T09:09:03.64425954Z,3,30,click","decoded_csv":{"result":"click","user":"3","time":"2023-03-28T09:09:03.64425954Z","page":"30"}} {"log":"2023-03-28T09:09:03.644317046Z,1,20,view","decoded_csv":{"result":"view","user":"1","time":"2023-03-28T09:09:03.644317046Z","page":"20"}} {"log":"2023-03-28T09:09:10.64447719Z,2,50,purchase","decoded_csv":{"result":"purchase","user":"2","time":"2023-03-28T09:09:10.64447719Z","page":"50"}} {"log":"2023-03-28T09:09:17.644810963Z,2,10,view","decoded_csv":{"result":"view","user":"2","time":"2023-03-28T09:09:17.644810963Z","page":"10"}} {"log":"2023-03-28T09:09:20.644994805Z,1,50,view","decoded_csv":{"result":"view","user":"1","time":"2023-03-28T09:09:20.644994805Z","page":"50"}} ``` This rule copied each row of CSV data to the `log` key and assigned each row's comma-separated values to a series of key-value pairs in `decoded_csv`. ### With parsed headers If you enable the **Parse header** setting, CSV data becomes a series of key-value pairs with keys based on the parsed header row. For example, given this sample CSV data: ```text theme={null} timestamp,user_id,page_id,action 2023-03-28T09:08:41.64283645Z,3,30,purchase 2023-03-28T09:08:42.643343109Z,4,10,purchase 2023-03-28T09:08:48.643600498Z,1,50,click 2023-03-28T09:08:50.643773688Z,5,40,purchase 2023-03-28T09:08:51.643932272Z,1,30,purchase 2023-03-28T09:08:56.644080944Z,2,40,click 2023-03-28T09:09:03.64425954Z,3,30,click 2023-03-28T09:09:03.644317046Z,1,20,view 2023-03-28T09:09:10.64447719Z,2,50,purchase 2023-03-28T09:09:17.644810963Z,2,10,view 2023-03-28T09:09:20.644994805Z,1,50,view ``` A processing rule with the **Source key** value `log`, the **Destination key** value `decoded_csv`, and the **Parse header** value enabled returns the following result: ```json theme={null} {"log":"2023-03-28T09:08:41.64283645Z,3,30,purchase","decoded_csv":{"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":"3","action":"purchase","page_id":"30"}} {"log":"2023-03-28T09:08:42.643343109Z,4,10,purchase","decoded_csv":{"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":"4","action":"purchase","page_id":"10"}} {"log":"2023-03-28T09:08:48.643600498Z,1,50,click","decoded_csv":{"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":"1","action":"click","page_id":"50"}} {"log":"2023-03-28T09:08:50.643773688Z,5,40,purchase","decoded_csv":{"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":"5","action":"purchase","page_id":"40"}} {"log":"2023-03-28T09:08:51.643932272Z,1,30,purchase","decoded_csv":{"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":"1","action":"purchase","page_id":"30"}} {"log":"2023-03-28T09:08:56.644080944Z,2,40,click","decoded_csv":{"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":"2","action":"click","page_id":"40"}} {"log":"2023-03-28T09:09:03.64425954Z,3,30,click","decoded_csv":{"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":"3","action":"click","page_id":"30"}} {"log":"2023-03-28T09:09:03.644317046Z,1,20,view","decoded_csv":{"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":"1","action":"view","page_id":"20"}} {"log":"2023-03-28T09:09:10.64447719Z,2,50,purchase","decoded_csv":{"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":"2","action":"purchase","page_id":"50"}} {"log":"2023-03-28T09:09:17.644810963Z,2,10,view","decoded_csv":{"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":"2","action":"view","page_id":"10"}} {"log":"2023-03-28T09:09:20.644994805Z,1,50,view","decoded_csv":{"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":"1","action":"view","page_id":"50"}} ``` This rule treated the first row of CSV data as a set of headers, then created a series of key-value pairs from subsequent rows by using header names as keys. This rule also copied each row of CSV data to the `log` key, except for the row that contained header names. ## Related rules For a processing rule with the opposite effect, see [encode CSV](/ingest/pipeline/processing-rules/encode-csv). # Decode JSON Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/decode-json Learn about the decode JSON processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The decode JSON [processing rule](/ingest/pipeline/processing-rules) transforms an escaped JSON string into a structured JSON object. The resulting output includes both the original string and a new JSON object assigned to the key of your choosing. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | | **Source key** | `src` | Required. The key that contains the escaped JSON string to transform. | `log` | | **Destination key** | `dst` | Required. The key to store your transformed JSON object. If a key with this name already exists, the decode JSON rule will overwrite it. | `decoded_json` | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Example Using the decode JSON rule lets you extract embedded data from a string and turn it into parsable key-value pairs. You can then use these key-value pairs in other processing rules or for general storage and analysis. For example, given the following sample website log data: ```json theme={null} {"log": "{\"timestamp\":\"2023-03-28T09:08:41.64283645Z\",\"user_id\":3,\"page_id\":30,\"action\":\"purchase\"}\n"} {"log": "{\"timestamp\":\"2023-03-28T09:08:42.643343109Z\",\"user_id\":4,\"page_id\":10,\"action\":\"purchase\"}\n"} {"log": "{\"timestamp\":\"2023-03-28T09:08:48.643600498Z\",\"user_id\":1,\"page_id\":50,\"action\":\"click\"}\n"} {"log": "{\"timestamp\":\"2023-03-28T09:08:50.643773688Z\",\"user_id\":5,\"page_id\":40,\"action\":\"purchase\"}\n"} {"log": "{\"timestamp\":\"2023-03-28T09:08:51.643932272Z\",\"user_id\":1,\"page_id\":30,\"action\":\"purchase\"}\n"} {"log": "{\"timestamp\":\"2023-03-28T09:08:56.644080944Z\",\"user_id\":2,\"page_id\":40,\"action\":\"click\"}\n"} {"log": "{\"timestamp\":\"2023-03-28T09:09:03.64425954Z\",\"user_id\":3,\"page_id\":30,\"action\":\"click\"}\n"} {"log": "{\"timestamp\":\"2023-03-28T09:09:03.644317046Z\",\"user_id\":1,\"page_id\":20,\"action\":\"view\"}\n"} {"log": "{\"timestamp\":\"2023-03-28T09:09:10.64447719Z\",\"user_id\":2,\"page_id\":50,\"action\":\"purchase\"}\n"} {"log": "{\"timestamp\":\"2023-03-28T09:09:17.644810963Z\",\"user_id\":2,\"page_id\":10,\"action\":\"view\"}\n"} {"log": "{\"timestamp\":\"2023-03-28T09:09:20.644994805Z\",\"user_id\":1,\"page_id\":50,\"action\":\"view\"}\n"} ``` A processing rule with the **Source key** value `log` and the **Destination key** value `output` returns the following result: ```json theme={null} {"output":{"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3},"log":"{\"timestamp\":\"2023-03-28T09:08:41.64283645Z\",\"user_id\":3,\"page_id\":30,\"action\":\"purchase\"}\n"} {"output":{"page_id":10,"action":"purchase","timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4},"log":"{\"timestamp\":\"2023-03-28T09:08:42.643343109Z\",\"user_id\":4,\"page_id\":10,\"action\":\"purchase\"}\n"} {"output":{"page_id":50,"action":"click","timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1},"log":"{\"timestamp\":\"2023-03-28T09:08:48.643600498Z\",\"user_id\":1,\"page_id\":50,\"action\":\"click\"}\n"} {"output":{"page_id":40,"action":"purchase","timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5},"log":"{\"timestamp\":\"2023-03-28T09:08:50.643773688Z\",\"user_id\":5,\"page_id\":40,\"action\":\"purchase\"}\n"} {"output":{"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1},"log":"{\"timestamp\":\"2023-03-28T09:08:51.643932272Z\",\"user_id\":1,\"page_id\":30,\"action\":\"purchase\"}\n"} {"output":{"page_id":40,"action":"click","timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2},"log":"{\"timestamp\":\"2023-03-28T09:08:56.644080944Z\",\"user_id\":2,\"page_id\":40,\"action\":\"click\"}\n"} {"output":{"page_id":30,"action":"click","timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3},"log":"{\"timestamp\":\"2023-03-28T09:09:03.64425954Z\",\"user_id\":3,\"page_id\":30,\"action\":\"click\"}\n"} {"output":{"page_id":20,"action":"view","timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1},"log":"{\"timestamp\":\"2023-03-28T09:09:03.644317046Z\",\"user_id\":1,\"page_id\":20,\"action\":\"view\"}\n"} {"output":{"page_id":50,"action":"purchase","timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2},"log":"{\"timestamp\":\"2023-03-28T09:09:10.64447719Z\",\"user_id\":2,\"page_id\":50,\"action\":\"purchase\"}\n"} {"output":{"page_id":10,"action":"view","timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2},"log":"{\"timestamp\":\"2023-03-28T09:09:17.644810963Z\",\"user_id\":2,\"page_id\":10,\"action\":\"view\"}\n"} {"output":{"page_id":50,"action":"view","timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1},"log":"{\"timestamp\":\"2023-03-28T09:09:20.644994805Z\",\"user_id\":1,\"page_id\":50,\"action\":\"view\"}\n"} ``` This rule retained the original `log` string and created a new JSON object called `output` that contains a structured version of the same key-value pairs that are embedded in `log`. ## Related rules For a processing rule with the opposite effect, see [encode JSON](/ingest/pipeline/processing-rules/encode-json). For a processing rule with a similar effect that works for any string, see [extract keys/values](/ingest/pipeline/processing-rules/extract-keys-values). # Deduplicate records Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/deduplicate-records Learn about the deduplicate records processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The deduplicate records [processing rule](/ingest/pipeline/processing-rules) looks for any records that contain identical key-value data during a specified time frame, then removes all but the first occurrence of those records within that time frame. When the deduplicate records rule waits for data to accumulate during the specified time window, the pipeline [buffers](/ingest/pipeline/v2/configure/backpressure) that data. Increasing the value of the **Time window** parameter also increases the memory load on your pipeline. For example, if 100,000 records pass through your pipeline during the specified time period, and those records are 1 kB each, the deduplicate records rule will add approximately 100 MB of memory load. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | --------------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | | **Time window** | `window` | Required. How long to wait (in seconds) for data to accumulate in your pipeline before searching for duplicate records. For example, for a window length of `5`, two records with an identical key-value pair are considered duplicates if they both occur within the same five-second period, but not if they occur within the same 10-second period. | *none* | | **Select key** | `key` | Required. The key to use in your comparison. If multiple records have the same value assigned to this key, this rule removes all but the earliest record to contain that key-value pair within the specified time frame. You can also use [record accessor syntax](/ingest/pipeline/processing-rules#record-accessor-syntax) to reference keys nested within another nested object. | *none* | | **Ignore records without key** checkbox | `skipMissing` | If selected, skips any records that don't contain your specified **Select key**. Chronosphere recommends selecting this checkbox to prevent processing errors. | Selected / `true` | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Example Using the deduplicate records rule lets you remove redundant information from your pipeline and reduce the amount of data that reaches your backend. For example, given the following sample log data: ```json lines theme={null} {"message": "All endpoints are functional."} {"message": "All endpoints are functional."} {"message": "All endpoints are functional."} {"message": "All endpoints are functional."} {"message": "The /purchase endpoint is unavailable."} {"message": "The /purchase endpoint is unavailable."} {"message": "The /purchase endpoint is unavailable."} {"message": "The /purchase endpoint is partly unavailable."} {"message": "The /purchase endpoint has been reset."} {"message": "All endpoints are functional."} {"message": "All endpoints are functional."} ``` A processing rule with the **Time window** value `5` and the **Source key** value `message` returns the following result: ```json theme={null} {"message":"All endpoints are functional."} {"message":"The /purchase endpoint is unavailable."} {"message":"The /purchase endpoint is partly unavailable."} {"message":"The /purchase endpoint has been reset."} {"message":"All endpoints are functional."} ``` This rule removed all but the first instance of any logs with identical `message` values that appeared within the specified time frame. Because more than five seconds elapsed between the value `All endpoints are functional` on line 1 and the same value on line 10, this rule retained both the log on line 1 and the log on line 10. However, since fewer than five seconds elapsed between the value `All endpoints are functional` on line 10 and the same value on line 11, this rule removed the log on line 11. # Delete key Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/delete-key Learn about the delete key processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The delete keys [processing rule](/ingest/pipeline/processing-rules) deletes a specified key and its associated value from every record that passes through your pipeline. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ----------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Key** | `key` | Required. The key to delete from your records. You can use [record accessor syntax](/ingest/pipeline/processing-rules#record-accessor-syntax) to reference keys nested within another object. | *none* | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Example Using the block keys rule lets you pare down telemetry data by eliminating key-value pairs you don't want to keep. For example, given the following sample website log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"} ``` A processing rule with the **Key** value `user_id` returns the following result: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","action":"purchase","page_id":30} {"timestamp":"2023-03-28T09:08:42.643343109Z","action":"purchase","page_id":10} {"timestamp":"2023-03-28T09:08:48.643600498Z","action":"click","page_id":50} {"timestamp":"2023-03-28T09:08:50.643773688Z","action":"purchase","page_id":40} {"timestamp":"2023-03-28T09:08:51.643932272Z","action":"purchase","page_id":30} {"timestamp":"2023-03-28T09:08:56.644080944Z","action":"click","page_id":40} {"timestamp":"2023-03-28T09:09:03.64425954Z","action":"click","page_id":30} {"timestamp":"2023-03-28T09:09:03.644317046Z","action":"view","page_id":20} {"timestamp":"2023-03-28T09:09:10.64447719Z","action":"purchase","page_id":50} {"timestamp":"2023-03-28T09:09:17.644810963Z","action":"view","page_id":10} {"timestamp":"2023-03-28T09:09:20.644994805Z","action":"view","page_id":50} ``` This rule removed the `user_id` key and its associated values from each log entry. # Encode CSV Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/encode-csv Learn about the encode CSV processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The encode CSV [processing rule](/ingest/pipeline/processing-rules) transforms log data from JSON to CSV format. The resulting output includes both the original JSON data and a string of comma-separated values assigned to a key you specify. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | **Source key** | `src` | Required. The key of the JSON data to transform. This JSON data must be either an array of strings or an object that contains key-value pairs. | *none* | | **Destination key** | `dst` | Required. The key to store your string of comma-separated values. If a key with this name already exists, the encode CSV rule will overwrite it. | `encoded_csv` | | **Header** | `header` | The header row for your comma-separated values. If **Source key** is a JSON object that contains key-value pairs, set this value to a list of the names of each key. Items in this list must be separated by commas and should match the name and order that the keys appear within their original JSON object. | *none* | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Examples Using the encode CSV rule lets you turn JSON data into comma-separated values that can be fed into databases and spreadsheet software. ### Array of strings If your JSON data is an array of strings, don't use the **Header** parameter. For example, given this sample JSON data: ```json theme={null} {"ingredients":["pasta","tomato","basil","olive oil","parmesan"]} {"ingredients":["potatoes","butter","salt","chives","sour cream","bacon bits","shredded cheese"]} {"ingredients":["cereal","milk"]} {"ingredients":["bread","mayo","mustard","lettuce","tomato","deli meat","pickle"]} ``` A processing rule with the **Source key** value `ingredients` and the **Destination key** value `encoded_csv` returns the following result: ```json theme={null} {"encoded_csv":"pasta,tomato,basil,olive oil,parmesan\n","ingredients":["pasta","tomato","basil","olive oil","parmesan"]} {"encoded_csv":"potatoes,butter,salt,chives,sour cream,bacon bits,shredded cheese\n","ingredients":["potatoes","butter","salt","chives","sour cream","bacon bits","shredded cheese"]} {"encoded_csv":"cereal,milk\n","ingredients":["cereal","milk"]} {"encoded_csv":"bread,mayo,mustard,lettuce,tomato,deli meat,pickle\n","ingredients":["bread","mayo","mustard","lettuce","tomato","deli meat","pickle"]} ``` This rule retained the original `ingredients` array and created a new key called `encoded_csv` to store the converted comma-separated values. ### Key-value pairs If your JSON data is an object that contains key-value pairs, you must use the **Header** parameter. For example, given this sample website log data: ```json theme={null} {"log":{"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":"3","action":"purchase","page_id":"30"}} {"log":{"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":"4","action":"purchase","page_id":"10"}} {"log":{"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":"1","action":"click","page_id":"50"}} {"log":{"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":"5","action":"purchase","page_id":"40"}} {"log":{"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":"1","action":"purchase","page_id":"30"}} {"log":{"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":"2","action":"click","page_id":"40"}} {"log":{"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":"3","action":"click","page_id":"30"}} {"log":{"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":"1","action":"view","page_id":"20"}} {"log":{"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":"2","action":"purchase","page_id":"50"}} {"log":{"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":"2","action":"view","page_id":"10"}} {"log":{"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":"1","action":"view","page_id":"50"}} ``` A processing rule with the **Source key** value `log`, the **Destination key** value `encoded_csv`, and the **Header** value `timestamp,user_id,action,page_id` returns the following result: ```json theme={null} {"log":{"page_id":"30","action":"purchase","timestamp":"2023-03-28T09:08:41.64283645Z","user_id":"3"},"encoded_csv":"timestamp,user_id,action,page_id\n2023-03-28T09:08:41.64283645Z,3,purchase,30\n"} {"log":{"page_id":"10","action":"purchase","timestamp":"2023-03-28T09:08:42.643343109Z","user_id":"4"},"encoded_csv":"2023-03-28T09:08:42.643343109Z,4,purchase,10\n"} {"log":{"page_id":"50","action":"click","timestamp":"2023-03-28T09:08:48.643600498Z","user_id":"1"},"encoded_csv":"2023-03-28T09:08:48.643600498Z,1,click,50\n"} {"log":{"page_id":"40","action":"purchase","timestamp":"2023-03-28T09:08:50.643773688Z","user_id":"5"},"encoded_csv":"2023-03-28T09:08:50.643773688Z,5,purchase,40\n"} {"log":{"page_id":"30","action":"purchase","timestamp":"2023-03-28T09:08:51.643932272Z","user_id":"1"},"encoded_csv":"2023-03-28T09:08:51.643932272Z,1,purchase,30\n"} {"log":{"page_id":"40","action":"click","timestamp":"2023-03-28T09:08:56.644080944Z","user_id":"2"},"encoded_csv":"2023-03-28T09:08:56.644080944Z,2,click,40\n"} {"log":{"page_id":"30","action":"click","timestamp":"2023-03-28T09:09:03.64425954Z","user_id":"3"},"encoded_csv":"2023-03-28T09:09:03.64425954Z,3,click,30\n"} {"log":{"page_id":"20","action":"view","timestamp":"2023-03-28T09:09:03.644317046Z","user_id":"1"},"encoded_csv":"2023-03-28T09:09:03.644317046Z,1,view,20\n"} {"log":{"page_id":"50","action":"purchase","timestamp":"2023-03-28T09:09:10.64447719Z","user_id":"2"},"encoded_csv":"2023-03-28T09:09:10.64447719Z,2,purchase,50\n"} {"log":{"page_id":"10","action":"view","timestamp":"2023-03-28T09:09:17.644810963Z","user_id":"2"},"encoded_csv":"2023-03-28T09:09:17.644810963Z,2,view,10\n"} {"log":{"page_id":"50","action":"view","timestamp":"2023-03-28T09:09:20.644994805Z","user_id":"1"},"encoded_csv":"2023-03-28T09:09:20.644994805Z,1,view,50\n"} ``` This rule retained the original `log` object and created a new key called `encoded_csv` to store the converted comma-separated values. Additionally, the first instance of `encoded_csv` contains a header row, which is also separated by commas. ## Related rules For a processing rule with the opposite effect, see [decode CSV](/ingest/pipeline/processing-rules/decode-csv). # Encode JSON Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/encode-json Learn about the encode JSON processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The encode JSON [processing rule](/ingest/pipeline/processing-rules) transforms a JSON object into an escaped string. The resulting output includes both the original JSON object and a new string assigned to a key you specify. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | | **Source key** | `src` | Required. The key that contains the JSON object to transform. | *none* | | **Destination key** | `dst` | Required. The key to store your escaped JSON string. If a key with this name already exists, the encode JSON rule will overwrite it. | `encoded_json` | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Example Using the encode JSON rule lets you condense a structured object into a single, condensed value. For example, given the following sample website log data: ```json theme={null} {"event":{"timestamp":"2023-03-28T09:08:41.64283645Z","page_id":30,"user_id":3,"action":"purchase"}} {"event":{"timestamp":"2023-03-28T09:08:42.643343109Z","page_id":10,"user_id":4,"action":"purchase"}} {"event":{"timestamp":"2023-03-28T09:08:48.643600498Z","page_id":50,"user_id":1,"action":"click"}} {"event":{"timestamp":"2023-03-28T09:08:50.643773688Z","page_id":40,"user_id":5,"action":"purchase"}} {"event":{"timestamp":"2023-03-28T09:08:51.643932272Z","page_id":30,"user_id":1,"action":"purchase"}} {"event":{"timestamp":"2023-03-28T09:08:56.644080944Z","page_id":40,"user_id":2,"action":"click"}} {"event":{"timestamp":"2023-03-28T09:09:03.64425954Z","page_id":30,"user_id":3,"action":"click"}} {"event":{"timestamp":"2023-03-28T09:09:03.644317046Z","page_id":20,"user_id":1,"action":"view"}} {"event":{"timestamp":"2023-03-28T09:09:10.64447719Z","page_id":50,"user_id":2,"action":"purchase"}} {"event":{"timestamp":"2023-03-28T09:09:17.644810963Z","page_id":10,"user_id":2,"action":"view"}} {"event":{"timestamp":"2023-03-28T09:09:20.644994805Z","page_id":50,"user_id":1,"action":"view"}} ``` A processing rule with the **Source key** value `event` and the **Destination key** value `stored_string` returns the following result: ```json theme={null} {"stored_string":"{\"action\":\"purchase\",\"page_id\":30,\"timestamp\":\"2023-03-28T09:08:41.64283645Z\",\"user_id\":3}","event":{"page_id":30,"user_id":3,"action":"purchase","timestamp":"2023-03-28T09:08:41.64283645Z"}} {"stored_string":"{\"action\":\"purchase\",\"page_id\":10,\"timestamp\":\"2023-03-28T09:08:42.643343109Z\",\"user_id\":4}","event":{"page_id":10,"user_id":4,"action":"purchase","timestamp":"2023-03-28T09:08:42.643343109Z"}} {"stored_string":"{\"action\":\"click\",\"page_id\":50,\"timestamp\":\"2023-03-28T09:08:48.643600498Z\",\"user_id\":1}","event":{"page_id":50,"user_id":1,"action":"click","timestamp":"2023-03-28T09:08:48.643600498Z"}} {"stored_string":"{\"action\":\"purchase\",\"page_id\":40,\"timestamp\":\"2023-03-28T09:08:50.643773688Z\",\"user_id\":5}","event":{"page_id":40,"user_id":5,"action":"purchase","timestamp":"2023-03-28T09:08:50.643773688Z"}} {"stored_string":"{\"action\":\"purchase\",\"page_id\":30,\"timestamp\":\"2023-03-28T09:08:51.643932272Z\",\"user_id\":1}","event":{"page_id":30,"user_id":1,"action":"purchase","timestamp":"2023-03-28T09:08:51.643932272Z"}} {"stored_string":"{\"action\":\"click\",\"page_id\":40,\"timestamp\":\"2023-03-28T09:08:56.644080944Z\",\"user_id\":2}","event":{"page_id":40,"user_id":2,"action":"click","timestamp":"2023-03-28T09:08:56.644080944Z"}} {"stored_string":"{\"action\":\"click\",\"page_id\":30,\"timestamp\":\"2023-03-28T09:09:03.64425954Z\",\"user_id\":3}","event":{"page_id":30,"user_id":3,"action":"click","timestamp":"2023-03-28T09:09:03.64425954Z"}} {"stored_string":"{\"action\":\"view\",\"page_id\":20,\"timestamp\":\"2023-03-28T09:09:03.644317046Z\",\"user_id\":1}","event":{"page_id":20,"user_id":1,"action":"view","timestamp":"2023-03-28T09:09:03.644317046Z"}} {"stored_string":"{\"action\":\"purchase\",\"page_id\":50,\"timestamp\":\"2023-03-28T09:09:10.64447719Z\",\"user_id\":2}","event":{"page_id":50,"user_id":2,"action":"purchase","timestamp":"2023-03-28T09:09:10.64447719Z"}} {"stored_string":"{\"action\":\"view\",\"page_id\":10,\"timestamp\":\"2023-03-28T09:09:17.644810963Z\",\"user_id\":2}","event":{"page_id":10,"user_id":2,"action":"view","timestamp":"2023-03-28T09:09:17.644810963Z"}} {"stored_string":"{\"action\":\"view\",\"page_id\":50,\"timestamp\":\"2023-03-28T09:09:20.644994805Z\",\"user_id\":1}","event":{"page_id":50,"user_id":1,"action":"view","timestamp":"2023-03-28T09:09:20.644994805Z"}} ``` This rule retained the original `event` object and created a new key called `stored_string` to store the same key-value pairs as an embedded string. ## Related rules For a processing rule with the opposite effect, see [decode JSON](/ingest/pipeline/processing-rules/decode-json). # Extract keys/values Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/extract-keys-values Learn about the extract keys/values processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The extract keys/values [processing rule](/ingest/pipeline/processing-rules) uses a regular expression to search for key-value pairs inside a string, then creates a structured object to store those key-value pairs. Additionally, the resulting output always includes a string of the original unstructured data. The resulting output includes both the original JSON data and a string of comma-separated values assigned to a key you specify. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Source key** | `src` | Required. The key whose value contains key-value pairs to extract. | *none* | | **Destination key** | `dst` | Required. The key of the object to store your structured key-value pairs. If you specify the name of an existing key, the original value of the key is overwritten. | *none* | | **Regex** | `regex` | Required. The regular expression for extracting key-value pairs from the value of **Source key**. This expression must have two capture groups: the result of the first capture group becomes the name of a key, and the result of the second capture group becomes that key's value. | *none* | | **Regex engine** | `regexEngine` | Required. The [engine](/ingest/pipeline/processing-rules#regex-engines) to parse your regular expression. Accepted values: `GNU`, `Oniguruma`, `PCRE2`, `POSIX`, `TRE`. | `PCRE2` | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Example Using the extract keys/values rule lets you extract embedded data from a string and turn it into parsable key-value pairs. You can then use these key-value pairs in other processing rules or for general storage and analysis. For example, given the following sample website log data: ```json theme={null} {"log": "user_id:3,page_id:30,action:purchase"} {"log": "user_id:4,page_id:10,action:purchase"} {"log": "user_id:1,page_id:50,action:click"} {"log": "user_id:5,page_id:40,action:purchase"} {"log": "user_id:1,page_id:30,action:purchase"} {"log": "user_id:2,page_id:40,action:click"} {"log": "user_id:3,page_id:30,action:click"} {"log": "user_id:1,page_id:20,action:view"} {"log": "user_id:2,page_id:50,action:purchase"} {"log": "user_id:2,page_id:10,action:view"} {"log": "user_id:1,page_id:50,action:view"} ``` A processing rule with the **Source key** value `log`, the **Destination key** value `extracted`, the **Regex** value `(\w+):(\w+)`, and the **Regex engine** value `PCRE2` returns the following result: ```json theme={null} {"extracted":{"action":"purchase","user_id":"3","page_id":"30"},"log":"user_id:3,page_id:30,action:purchase"} {"extracted":{"action":"purchase","user_id":"4","page_id":"10"},"log":"user_id:4,page_id:10,action:purchase"} {"extracted":{"action":"click","user_id":"1","page_id":"50"},"log":"user_id:1,page_id:50,action:click"} {"extracted":{"action":"purchase","user_id":"5","page_id":"40"},"log":"user_id:5,page_id:40,action:purchase"} {"extracted":{"action":"purchase","user_id":"1","page_id":"30"},"log":"user_id:1,page_id:30,action:purchase"} {"extracted":{"action":"click","user_id":"2","page_id":"40"},"log":"user_id:2,page_id:40,action:click"} {"extracted":{"action":"click","user_id":"3","page_id":"30"},"log":"user_id:3,page_id:30,action:click"} {"extracted":{"action":"view","user_id":"1","page_id":"20"},"log":"user_id:1,page_id:20,action:view"} {"extracted":{"action":"purchase","user_id":"2","page_id":"50"},"log":"user_id:2,page_id:50,action:purchase"} {"extracted":{"action":"view","user_id":"2","page_id":"10"},"log":"user_id:2,page_id:10,action:view"} {"extracted":{"action":"view","user_id":"1","page_id":"50"},"log":"user_id:1,page_id:50,action:view"} ``` This rule extracted key-value pairs from the string stored in the `log` key and stored those key-value pairs in a new structured object named `extracted`. ## Related rules For a processing rule that performs a similar operation on escaped JSON strings, see [decode JSON](/ingest/pipeline/processing-rules/decode-json). For a processing rule that performs a similar operation on data not already formatted as key-value pairs, see [parse](/ingest/pipeline/processing-rules/parse). # Flatten subrecord Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/flatten-subrecord Learn about the flatten subrecord processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The flatten subrecord [processing rule](/ingest/pipeline/processing-rules) uses regular expressions to search for key-value pairs inside a JSON object, then either moves or copies any applicable key-value pairs to the top level of the record. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | -------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Key** | `key` | Required. The key of the object that contains key-value pairs to move or copy. You can also use [record accessor syntax](/ingest/pipeline/processing-rules#record-accessor-syntax) to reference keys nested in another nested object. | *none* | | **Regex** | `regex` | Required. The regular expression that determines which key-value pairs to move or copy. A value of `^.+$` matches all key-value pairs contained by the specified object. | `^.+$` | | **Replacement key** | `keyReplacement` | Required. The regular expression that sets the naming pattern for keys that are moved or copied. A value of `%1` preserves all keys' original names. | `%1` | | **Keep original** checkbox | `keepOrig` | If selected, preserves the object specified in **Key**. If unselected, the flatten subrecord rule moves any applicable key-value pairs to the top level of the record, and then removes the object specified in **Key**. | Not selected | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Examples Using the flatten subrecord rule lets you restructure your telemetry data by pulling key-value pairs out of a nested object and storing them as flat data at the record's root. ### Move key-value pairs For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"account":{"vip":"no","id":3},"device":"mobile"},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"account":{"vip":"yes","id":4},"device":"desktop"},"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"account":{"vip":"yes","id":5},"device":"mobile"},"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"account":{"vip":"no","id":1},"device":"other"},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"account":{"vip":"yes","id":2},"device":"mobile"},"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"account":{"vip":"no","id":3},"device":"desktop"},"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"account":{"vip":"yes","id":2},"device":"desktop"},"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"account":{"vip":"yes","id":2},"device":"other"},"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"account":{"vip":"no","id":1},"device":"mobile"},"page_id":50,"action":"view"} ``` A processing rule with the **Key** value `$user.account`, the **Regex** value `id`, and the **Replacement key** value `user.%1` returns the following result: ```json theme={null} {"user.id":3,"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"device":"mobile"},"action":"purchase","page_id":30} {"user.id":4,"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"device":"desktop"},"action":"purchase","page_id":10} {"user.id":1,"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"device":"desktop"},"action":"click","page_id":50} {"user.id":5,"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"device":"mobile"},"action":"purchase","page_id":40} {"user.id":1,"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"device":"other"},"action":"purchase","page_id":30} {"user.id":2,"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"device":"mobile"},"action":"click","page_id":40} {"user.id":3,"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"device":"desktop"},"action":"click","page_id":30} {"user.id":1,"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"device":"desktop"},"action":"view","page_id":20} {"user.id":2,"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"device":"desktop"},"action":"purchase","page_id":50} {"user.id":2,"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"device":"other"},"action":"view","page_id":10} {"user.id":1,"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"device":"mobile"},"action":"view","page_id":50} ``` This rule searched for applicable key-value pairs inside the `account` object, moved `id` up into a new key named `user.id` at the record's top level, and then removed `account` and the key-value pairs it originally contained. ### Copy key-value pairs You can also use the **Keep original** setting to copy key-value pairs out of an object instead of moving them. For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"account":{"vip":"no","id":3},"device":"mobile"},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"account":{"vip":"yes","id":4},"device":"desktop"},"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"account":{"vip":"yes","id":5},"device":"mobile"},"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"account":{"vip":"no","id":1},"device":"other"},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"account":{"vip":"yes","id":2},"device":"mobile"},"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"account":{"vip":"no","id":3},"device":"desktop"},"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"account":{"vip":"yes","id":2},"device":"desktop"},"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"account":{"vip":"yes","id":2},"device":"other"},"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"account":{"vip":"no","id":1},"device":"mobile"},"page_id":50,"action":"view"} ``` A processing rule with the **Key** value `$user.account`, the **Regex** value `id`, the **Replacement key** value `user.%1`, and the **Keep original** setting enabled returns the following result: ```json theme={null} {"user.id":3,"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"device":"mobile","account":{"vip":"no","id":3}},"action":"purchase","page_id":30} {"user.id":4,"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"device":"desktop","account":{"vip":"yes","id":4}},"action":"purchase","page_id":10} {"user.id":1,"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"device":"desktop","account":{"vip":"no","id":1}},"action":"click","page_id":50} {"user.id":5,"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"device":"mobile","account":{"vip":"yes","id":5}},"action":"purchase","page_id":40} {"user.id":1,"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"device":"other","account":{"vip":"no","id":1}},"action":"purchase","page_id":30} {"user.id":2,"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"device":"mobile","account":{"vip":"yes","id":2}},"action":"click","page_id":40} {"user.id":3,"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"device":"desktop","account":{"vip":"no","id":3}},"action":"click","page_id":30} {"user.id":1,"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"device":"desktop","account":{"vip":"no","id":1}},"action":"view","page_id":20} {"user.id":2,"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"device":"desktop","account":{"vip":"yes","id":2}},"action":"purchase","page_id":50} {"user.id":2,"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"device":"other","account":{"vip":"yes","id":2}},"action":"view","page_id":10} {"user.id":1,"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"device":"mobile","account":{"vip":"no","id":1}},"action":"view","page_id":50} ``` This rule searched for applicable key-value pairs inside the `account` object, copied `id` into a new key named `user.id` at the record's top level, but did not remove `account` and the key-value pairs it originally contained. ## Related rules For a processing rule with a similar effect, see [lift submap](/ingest/pipeline/processing-rules/lift-submap). For a processing rule with the opposite effect, see [nest keys](/ingest/pipeline/processing-rules/nest-keys). # Hash key Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/hash-key Learn about the hash key processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The hash key [processing rule](/ingest/pipeline/processing-rules) copies the value of a specified source key, hashes that value, then stores the hashed copy in a specified destination key. Unless you configure this rule to overwrite the specified source key, the resulting output will still include the source key and its original, un-hashed value. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ---------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | | **Source key** | `src` | Required. The key whose value to hash. The value of this key must be a string and not a number, Boolean, array, or object. | *none* | | **Destination key** | `dst` | The key that stores the hashed value of **Source key**. If you specify the name of an existing key, the hash keys rule overwrites the original value of that key. If you don't specify a key name, this rule creates a new key with the name `_SOURCE_HASH_`, where `SOURCE` is the name of your **Source Key** and `HASH` is the name of your **Hash algorithm**. | *none* | | **Encoding algorithm** | `encoding` | Required. The scheme to store your hashed value. Accepted values: `Base64`, `Binary`, `Hexadecimal`. | `Hexadecimal` | | **Hash algorithm** | `algo` | Required. The algorithm to hash your value. Accepted values: `MD5`, `SHA256`. | `SHA256` | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Example Using the hash keys rule lets you store values in hashed format, which can help validate the authenticity and integrity of your data. For example, given the following sample website log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"} ``` A processing rule with the **Source key** value `action`, the **Encoding algorithm** value `Hexadecimal`, and the **Hash algorithm** value `SHA256` returns the following result: ```json theme={null} {"user_id":3,"page_id":30,"action":"purchase","action_sha256":"7048dc8aa7663e713097535171b4ed72ab015705a885de9636e66d4b1fa18804","timestamp":"2023-03-28T09:08:41.64283645Z"} {"user_id":4,"page_id":10,"action":"purchase","action_sha256":"7048dc8aa7663e713097535171b4ed72ab015705a885de9636e66d4b1fa18804","timestamp":"2023-03-28T09:08:42.643343109Z"} {"user_id":1,"page_id":50,"action":"click","action_sha256":"e5c7ffac26fed654fe62045898f55b551a0dc120badf3d116bcd364418f3ec16","timestamp":"2023-03-28T09:08:48.643600498Z"} {"user_id":5,"page_id":40,"action":"purchase","action_sha256":"7048dc8aa7663e713097535171b4ed72ab015705a885de9636e66d4b1fa18804","timestamp":"2023-03-28T09:08:50.643773688Z"} {"user_id":1,"page_id":30,"action":"purchase","action_sha256":"7048dc8aa7663e713097535171b4ed72ab015705a885de9636e66d4b1fa18804","timestamp":"2023-03-28T09:08:51.643932272Z"} {"user_id":2,"page_id":40,"action":"click","action_sha256":"e5c7ffac26fed654fe62045898f55b551a0dc120badf3d116bcd364418f3ec16","timestamp":"2023-03-28T09:08:56.644080944Z"} {"user_id":3,"page_id":30,"action":"click","action_sha256":"e5c7ffac26fed654fe62045898f55b551a0dc120badf3d116bcd364418f3ec16","timestamp":"2023-03-28T09:09:03.64425954Z"} {"user_id":1,"page_id":20,"action":"view","action_sha256":"2bcb43cbc8f6b7ef66331532881143fcbae60a879db3a8fb853f645bb24c2b3c","timestamp":"2023-03-28T09:09:03.644317046Z"} {"user_id":2,"page_id":50,"action":"purchase","action_sha256":"7048dc8aa7663e713097535171b4ed72ab015705a885de9636e66d4b1fa18804","timestamp":"2023-03-28T09:09:10.64447719Z"} {"user_id":2,"page_id":10,"action":"view","action_sha256":"2bcb43cbc8f6b7ef66331532881143fcbae60a879db3a8fb853f645bb24c2b3c","timestamp":"2023-03-28T09:09:17.644810963Z"} {"user_id":1,"page_id":50,"action":"view","action_sha256":"2bcb43cbc8f6b7ef66331532881143fcbae60a879db3a8fb853f645bb24c2b3c","timestamp":"2023-03-28T09:09:20.644994805Z"} ``` This rule hashed the value of `action` from each log entry and stored the hashed value in a new key named `action_sha256`. # Join records Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/join-records Learn about the join records processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The join records [processing rule](/ingest/pipeline/processing-rules) combines values from multiple records into an array of values within a single record. To determine which values to combine, this rule checks the timestamp of each record, then combines values from records that occur within the same nanosecond. This timestamp is a piece of metadata that Chronosphere Telemetry Pipeline assigns to all records that pass through a pipeline, and is unrelated to any keys within the record itself. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ------------------------------------------------------------ | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | **Source key** | `key` | Required. The key that contains values to join in an array. The resulting array will be assigned to a key with the same name. | *none* | | **Skip records where the specified key is missing** checkbox | `skipMissing` | Indicates whether to modify the record if the key isn't found. | Not selected | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Example Using the join records processing rule lets you restructure your telemetry data by turning multiple standalone events into a combined array. For example, given these sample logs: ```json theme={null} {"log":{"user_id":3,"action":"purchase"}} {"log":{"user_id":4,"action":"click"}} {"log":{"user_id":1,"action":"view"}} {"log":{"user_id":5,"action":"click"}} {"log":{"user_id":2,"action":"click"}} ``` A processing rule with the **Source key** value `log` returns the following result: ```json theme={null} {"log":[{"user_id":3,"action":"purchase"},{"user_id":4,"action":"click"},{"user_id":1,"action":"view"}]} {"log":[{"user_id":5,"action":"click"},{"user_id":2,"action":"click"}]} ``` This rule combined the `log` key from multiple standalone records into a series of unified `log` arrays. Because the first three events occurred within the same nanosecond, their `log` values were combined within one array, and because the fourth and fifth events occurred one nanosecond later, their `log` values were combined within another array. ## Related rules For a processing rule with the opposite effect, see [split record](/ingest/pipeline/processing-rules/split-record). # Lift submap Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/lift-submap Learn about the lift submap processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The lift submap [processing rule](/ingest/pipeline/processing-rules) uses regular expressions to search for key-value pairs inside a JSON object, then either moves or copies any applicable key-value pairs out of the JSON object and into a higher level of the record. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | -------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ | | **Key** | `key` | Required. The key of the object that contains key-value pairs to move or copy. You can also use [record accessor syntax](/ingest/pipeline/processing-rules#record-accessor-syntax) to reference keys nested within another nested object. | *none* | | **Regex** | `regex` | Required. The regular expression that determines which key-value pairs to move or copy. A value of `^.+$` matches all key-value pairs contained within the specified object. | `^.+$` | | **Regex engine** | `regexEngine` | Required. The [engine](/ingest/pipeline/processing-rules#regex-engines) to parse your regular expression. Accepted values: `GNU`, `Oniguruma`, `PCRE2`, `POSIX`, `TRE`. | `PCRE2` | | **Levels** | `levels` | Required. How many levels to lift the applicable key-value pairs. For example, the a value of `1` either moves or copies any applicable key-value pairs up one level into the parent object of **Key**. A value of `3` either moves or copies any applicable key-value pairs up three levels into the great-grandparent object of **Key**. | `1` | | **Replacement key** | `keyReplacement` | Required. The regular expression that sets the naming pattern for any keys that get moved or copied. A value of `%1` preserves the original names of all keys. | `%1` | | **Keep original** checkbox | `keepOrig` | If selected, preserves the object specified in **Key**. If unselected, moves any applicable key-value pairs into a higher level of the record, and then remove the object specified in **Key**. | Not selected | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Examples Using the lift submap rule lets you restructure your telemetry data by pulling key-value pairs out of a nested object and storing them as flat data in a higher level of the record. You can use this rule to simplify deeply nested data and store key-value pairs in more accessible locations. ### Move key-value pairs For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"account":{"vip":"no","id":3},"device":"mobile"},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"account":{"vip":"yes","id":4},"device":"desktop"},"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"account":{"vip":"yes","id":5},"device":"mobile"},"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"account":{"vip":"no","id":1},"device":"other"},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"account":{"vip":"yes","id":2},"device":"mobile"},"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"account":{"vip":"no","id":3},"device":"desktop"},"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"account":{"vip":"yes","id":2},"device":"desktop"},"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"account":{"vip":"yes","id":2},"device":"other"},"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"account":{"vip":"no","id":1},"device":"mobile"},"page_id":50,"action":"view"} ``` A processing rule with the **Key** value `$user.account`, the **Regex** value `id`, the **Levels** value `1`, and the **Replacement key** value `account.%1` returns the following result: ```json theme={null} {"user":{"account.id":3,"device":"mobile"},"timestamp":"2023-03-28T09:08:41.64283645Z","action":"purchase","page_id":30} {"user":{"account.id":4,"device":"desktop"},"timestamp":"2023-03-28T09:08:42.643343109Z","action":"purchase","page_id":10} {"user":{"account.id":1,"device":"desktop"},"timestamp":"2023-03-28T09:08:48.643600498Z","action":"click","page_id":50} {"user":{"account.id":5,"device":"mobile"},"timestamp":"2023-03-28T09:08:50.643773688Z","action":"purchase","page_id":40} {"user":{"account.id":1,"device":"other"},"timestamp":"2023-03-28T09:08:51.643932272Z","action":"purchase","page_id":30} {"user":{"account.id":2,"device":"mobile"},"timestamp":"2023-03-28T09:08:56.644080944Z","action":"click","page_id":40} {"user":{"account.id":3,"device":"desktop"},"timestamp":"2023-03-28T09:09:03.64425954Z","action":"click","page_id":30} {"user":{"account.id":1,"device":"desktop"},"timestamp":"2023-03-28T09:09:03.644317046Z","action":"view","page_id":20} {"user":{"account.id":2,"device":"desktop"},"timestamp":"2023-03-28T09:09:10.64447719Z","action":"purchase","page_id":50} {"user":{"account.id":2,"device":"other"},"timestamp":"2023-03-28T09:09:17.644810963Z","action":"view","page_id":10} {"user":{"account.id":1,"device":"mobile"},"timestamp":"2023-03-28T09:09:20.644994805Z","action":"view","page_id":50} ``` This rule searched for applicable key-value pairs inside the `account` object, moved `id` up one level into a new key named `account.id`, and then removed `account` and the key-value pairs it originally contained. ### Copy key-value pairs You can also use the **Keep original** setting to copy key-value pairs out of an object instead of moving them. For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"account":{"vip":"no","id":3},"device":"mobile"},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"account":{"vip":"yes","id":4},"device":"desktop"},"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"account":{"vip":"yes","id":5},"device":"mobile"},"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"account":{"vip":"no","id":1},"device":"other"},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"account":{"vip":"yes","id":2},"device":"mobile"},"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"account":{"vip":"no","id":3},"device":"desktop"},"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"account":{"vip":"yes","id":2},"device":"desktop"},"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"account":{"vip":"yes","id":2},"device":"other"},"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"account":{"vip":"no","id":1},"device":"mobile"},"page_id":50,"action":"view"} ``` A processing rule with the **Key** value `$user.account`, the **Regex** value `id`, the **Levels** value `1`, the **Replacement key** value `account.%1`, and the **Keep original** setting enabled returns the following result: ```json theme={null} {"user":{"account":{"id":3,"vip":"no"},"account.id":3,"device":"mobile"},"timestamp":"2023-03-28T09:08:41.64283645Z","action":"purchase","page_id":30} {"user":{"account":{"id":4,"vip":"yes"},"account.id":4,"device":"desktop"},"timestamp":"2023-03-28T09:08:42.643343109Z","action":"purchase","page_id":10} {"user":{"account":{"id":1,"vip":"no"},"account.id":1,"device":"desktop"},"timestamp":"2023-03-28T09:08:48.643600498Z","action":"click","page_id":50} {"user":{"account":{"id":5,"vip":"yes"},"account.id":5,"device":"mobile"},"timestamp":"2023-03-28T09:08:50.643773688Z","action":"purchase","page_id":40} {"user":{"account":{"id":1,"vip":"no"},"account.id":1,"device":"other"},"timestamp":"2023-03-28T09:08:51.643932272Z","action":"purchase","page_id":30} {"user":{"account":{"id":2,"vip":"yes"},"account.id":2,"device":"mobile"},"timestamp":"2023-03-28T09:08:56.644080944Z","action":"click","page_id":40} {"user":{"account":{"id":3,"vip":"no"},"account.id":3,"device":"desktop"},"timestamp":"2023-03-28T09:09:03.64425954Z","action":"click","page_id":30} {"user":{"account":{"id":1,"vip":"no"},"account.id":1,"device":"desktop"},"timestamp":"2023-03-28T09:09:03.644317046Z","action":"view","page_id":20} {"user":{"account":{"id":2,"vip":"yes"},"account.id":2,"device":"desktop"},"timestamp":"2023-03-28T09:09:10.64447719Z","action":"purchase","page_id":50} {"user":{"account":{"id":2,"vip":"yes"},"account.id":2,"device":"other"},"timestamp":"2023-03-28T09:09:17.644810963Z","action":"view","page_id":10} {"user":{"account":{"id":1,"vip":"no"},"account.id":1,"device":"mobile"},"timestamp":"2023-03-28T09:09:20.644994805Z","action":"view","page_id":50} ``` This rule searched for applicable key-value pairs inside the `account` object, moved `id` up one level into a new key named `account.id`, but did not remove `account` and the key-value pairs it originally contained. ## Related rules For a processing rule with a similar effect, see [flatten subrecord](/ingest/pipeline/processing-rules/flatten-subrecord). For a processing rule with the opposite effect, see [nest keys](/ingest/pipeline/processing-rules/nest-keys). # Multiline join Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/multiline-join Learn about the multiline join processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The multiline join [processing rule](/ingest/pipeline/processing-rules) combines multiple logs into a single log by looking for repeating patterns in log data. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ----------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | **Source key** | `src` | Required. The key that contains the log data to combine. This key's value must be a string, as the rule can't parse standard JSON objects, but it can parse [escaped JSON strings](/ingest/pipeline/processing-rules/encode-json). | `log` | | **Regex** | `regex` | Required. The regular expression that determines which logs to combine. When a log that contains a matching string is found, that log plus all subsequent logs are combined until another log is reached that also contains a matching string. The new matching log becomes the start of the next combined log, and the pattern repeats. | *none* | | **Regex engine** | `regexEngine` | Required. The [engine](/ingest/pipeline/processing-rules#regex-engines) to parse your regular expression. Accepted values: `GNU`, `Oniguruma`, `PCRE2`, `POSIX`, `TRE`. | `PCRE2` | | **Maximum lines** | `lineLimit` | Required. The maximum number of logs that can be combined into a single log. This setting is a failsafe designed to prevent the rule from falling into an endless loop if it can't find the specified **Regex** pattern after a certain number of logs. | `1000` | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Examples Using the multiline join processing rule lets you group related logs together. You can use this rule to combine data that was inadvertently split into multiple parts, like a stack trace where each line of the message was assigned to its own log. ### JSON logs For example, given this sample log data: ```json theme={null} {"inventory":"fruit:apricot"} {"inventory":"vegetable:artichoke"} {"inventory":"vegetable:asparagus"} {"inventory":"fruit:banana"} {"inventory":"vegetable:butternut_squash"} {"inventory":"fruit:cantaloupe"} {"inventory":"vegetable:cauliflower"} {"inventory":"vegetable:corn"} {"inventory":"vegetable:cucumber"} {"inventory":"fruit:durian"} {"inventory":"vegetable:daikon"} {"inventory":"fruit:elderberry"} {"inventory":"vegetable:edamame"} ``` A processing rule with the **Source key** value `inventory` and the **Regex** value `fruit` returns the following result: ```json theme={null} {"inventory":"fruit:apricot\nvegetable:artichoke\nvegetable:asparagus"} {"inventory":"fruit:banana\nvegetable:butternut_squash"} {"inventory":"fruit:cantaloupe\nvegetable:cauliflower\nvegetable:corn\nvegetable:cucumber"} {"inventory":"fruit:durian\nvegetable:daikon"} {"inventory":"fruit:elderberry\nvegetable:edamame"} ``` This rule searched for logs in the `inventory` key that contained the string `fruit`, then combined each of those logs with any subsequent logs that didn't contain the string `fruit`. After the rule found another log that contained the string `fruit`, it started over and repeated the same pattern. ### Raw logs To combine raw logs, always use the value `log` for **Source key**. You can also use the multiline join rule to combine raw logs. For example, given this sample log data: ```text theme={null} Exception in thread "main" java.lang.RuntimeException: at com.storefront.module.Checkout.paymentProcess(Checkout.java:71) at com.storefront.module.Checkout.billingAddress(Checkout.java:435) at com.storefront.module.Checkout.shippingAddress(Checkout.java:742) at com.storefront.module.Checkout.main(Checkout.java:6) Exception in thread "main" java.lang.NullPointerException: at com.storefront.module.Listing.productPhoto(Listing.java:84) at com.storefront.module.Listing.productColorway(Listing.java:219) Exception in thread "main" java.lang.RuntimeException: at com.storefront.module.Settings.country(Settings.java:149) at com.storefront.module.Settings.currencyType(Settings.java:736) at com.storefront.module.Settings.main(Checkout.java:11) ``` A processing rule with the **Source key** value `log` and the **Regex** value `exception` returns the following result: ```json theme={null} {"log":"Exception in thread \"main\" java.lang.RuntimeException:\n at com.storefront.module.Checkout.paymentProcess(Checkout.java:71)\n at com.storefront.module.Checkout.billingAddress(Checkout.java:435)\n at com.storefront.module.Checkout.shippingAddress(Checkout.java:742)\n at com.storefront.module.Checkout.main(Checkout.java:6)"} {"log":"Exception in thread \"main\" java.lang.NullPointerException:\n at com.storefront.module.Listing.productPhoto(Listing.java:84)\n at com.storefront.module.Listing.productColorway(Listing.java:219)"} {"log":"Exception in thread \"main\" java.lang.RuntimeException:\n at com.storefront.module.Settings.country(Settings.java:149)\n at com.storefront.module.Settings.currencyType(Settings.java:736)\n at com.storefront.module.Settings.main(Checkout.java:11)"} ``` Chronosphere Telemetry Pipeline assigned a new `log` key to each raw event. After that transformation, this rule searched for logs with the `log` key that contained the string `exception`, then combined each of those logs with any subsequent logs that didn't contain the string `exception`. After the rule found another log that contained the string `exception`, it started over and repeated the same pattern. # Nest keys Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/nest-keys Learn about the nest keys processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The nest keys [processing rule](/ingest/pipeline/processing-rules) moves the value of a specified source key into an object nested under a specified destination key. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------- | | **Source key** | `src` | Required. The key to nest under **Destination key**. This can be either a flat key-value pair or the key to an object. | *none* | | **Destination key** | `dst` | Required. The key of the object to store your nested data. If no matching key exists, create a new key and set its value accordingly. | *none* | | **Match case** checkbox | `matchCase` | Indicates whether the regular expression is case-sensitive. | Not selected | | **Regex engine** | `regexEngine` | Required. The [engine](/ingest/pipeline/processing-rules#regex-engines) to parse your regular expression. Accepted values: `GNU`, `Oniguruma`, `PCRE2`, `POSIX`, `TRE`. | `PCRE2` | | **Regex mode** checkbox | `regexMode` | Indicates whether to treat the value of **Source key** as a regular expression. | Not selected / `false` | | **Merge** checkbox | `merge` | If selected, and if **Destination key** is the key to an existing object that already contains data, the nest keys rule moves the new nested data into **Destination key** without overwriting its original data. However, if **Destination key** is a flat key-value pair, its value is overwritten and replaced with a new object. | Not selected | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## Examples Using the nest keys processing rule lets you restructure your telemetry data by storing key-value pairs inside a parent object. ### New object For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"} ``` A processing rule with the **Source key** value `user_id` and the **Destination key** value `account` returns the following result: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"account":{"user_id":3},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"account":{"user_id":4},"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"account":{"user_id":1},"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"account":{"user_id":5},"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"account":{"user_id":1},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"account":{"user_id":2},"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"account":{"user_id":3},"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"account":{"user_id":1},"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"account":{"user_id":2},"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"account":{"user_id":2},"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"account":{"user_id":1},"page_id":50,"action":"view"} ``` This rule moved the `user_id` key-value pair into a new object named `account`. ### Merge into existing object You can also use the **Merge** setting to nest data within an existing object. For example, given this sample log data: ```json theme={null} {"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"account":{"vip":"no","id":3},"device":"mobile"},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"account":{"vip":"yes","id":4},"device":"desktop"},"page_id":10,"action":"purchase"} {"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":50,"action":"click"} {"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"account":{"vip":"yes","id":5},"device":"mobile"},"page_id":40,"action":"purchase"} {"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"account":{"vip":"no","id":1},"device":"other"},"page_id":30,"action":"purchase"} {"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"account":{"vip":"yes","id":2},"device":"mobile"},"page_id":40,"action":"click"} {"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"account":{"vip":"no","id":3},"device":"desktop"},"page_id":30,"action":"click"} {"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":20,"action":"view"} {"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"account":{"vip":"yes","id":2},"device":"desktop"},"page_id":50,"action":"purchase"} {"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"account":{"vip":"yes","id":2},"device":"other"},"page_id":10,"action":"view"} {"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"account":{"vip":"no","id":1},"device":"mobile"},"page_id":50,"action":"view"} ``` A processing rule with the **Source key** value `action`, the **Destination key** value `user`, and the **Merge** setting enabled returns the following result: ```json theme={null} {"user":{"account":{"id":3,"vip":"no"},"action":"purchase","device":"mobile"},"timestamp":"2023-03-28T09:08:41.64283645Z","page_id":30,"action":"purchase"} {"user":{"account":{"id":4,"vip":"yes"},"action":"purchase","device":"desktop"},"timestamp":"2023-03-28T09:08:42.643343109Z","page_id":10,"action":"purchase"} {"user":{"account":{"id":1,"vip":"no"},"action":"click","device":"desktop"},"timestamp":"2023-03-28T09:08:48.643600498Z","page_id":50,"action":"click"} {"user":{"account":{"id":5,"vip":"yes"},"action":"purchase","device":"mobile"},"timestamp":"2023-03-28T09:08:50.643773688Z","page_id":40,"action":"purchase"} {"user":{"account":{"id":1,"vip":"no"},"action":"purchase","device":"other"},"timestamp":"2023-03-28T09:08:51.643932272Z","page_id":30,"action":"purchase"} {"user":{"account":{"id":2,"vip":"yes"},"action":"click","device":"mobile"},"timestamp":"2023-03-28T09:08:56.644080944Z","page_id":40,"action":"click"} {"user":{"account":{"id":3,"vip":"no"},"action":"click","device":"desktop"},"timestamp":"2023-03-28T09:09:03.64425954Z","page_id":30,"action":"click"} {"user":{"account":{"id":1,"vip":"no"},"action":"view","device":"desktop"},"timestamp":"2023-03-28T09:09:03.644317046Z","page_id":20,"action":"view"} {"user":{"account":{"id":2,"vip":"yes"},"action":"purchase","device":"desktop"},"timestamp":"2023-03-28T09:09:10.64447719Z","page_id":50,"action":"purchase"} {"user":{"account":{"id":2,"vip":"yes"},"action":"view","device":"other"},"timestamp":"2023-03-28T09:09:17.644810963Z","page_id":10,"action":"view"} {"user":{"account":{"id":1,"vip":"no"},"action":"view","device":"mobile"},"timestamp":"2023-03-28T09:09:20.644994805Z","page_id":50,"action":"view"} ``` This rule moved the `action` key-value pair into the `user` object, but kept the existing key-value pairs that were already stored in `user`. ## Related rules For a processing rule with the opposite effect, see [flatten subrecord](/ingest/pipeline/processing-rules/flatten-subrecord) or [lift submap](/ingest/pipeline/processing-rules/lift-submap). # Parse Source: https://docs.chronosphere.io/ingest/pipeline/processing-rules/parse Learn about the parse processing rule in Chronosphere Telemetry Pipeline. This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform. The parse [processing rule](/ingest/pipeline/processing-rules) uses a regular expression to search for values inside a string and to assign a key to each value, then stores those key-value pairs in a structured object. Additionally, the resulting output always includes the original string. ## Configuration parameters Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the **Name** column to describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files) use the items in the **Key** column as YAML keys. | Name | Key | Description | Default | | ------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | | **Source key** | `src` | Required. The key whose value contains data to parse. | *none* | | **Destination key** | `dst` | Required. The key of the object to store your structured key-value pairs. This rule can't overwrite an existing key, so this value must be a unique name within your telemetry data. | *none* | | **Regex** | `regex` | Required. The regular expression for extracting values from the value of **Source key** and assigning keys to those values. | *none* | | **Regex engine** | `regexEngine` | Required. The [engine](/ingest/pipeline/processing-rules#regex-engines) to parse your regular expression. Accepted values: `GNU`, `Oniguruma`, `PCRE2`, `POSIX`, `TRE`. | `PCRE2` | | **Comment** | `comment` | A custom note or description of the rule's function. This text is displayed next to the rule's name in the **Actions** list in the processing rules interface. | *none* | ## GPT generation Keep in mind that this feature is experimental, and that Chronosphere can't guarantee the accuracy or quality of generated scripts. To enable or disable GPT generation, see [Project settings](/ingest/pipeline/administer/project-settings#enable-or-disable-gpt-generation). The parse processing rule lets you generate regular expressions by using a generative pre-trained transformer (GPT). To use this feature, click **Generate Regular Expression With Gpt Experimental**. GPT generation uses any logs present in the **Input** field of the processing rules builder as the basis for its prompt. ## Example Using the parse processing rule lets you extract any data from a string and turn that data into parsable key-value pairs. You can then use these key-value pairs in other processing rules or for general storage and analysis. For example, given the following sample website logs: ```json theme={null} {"log":"198.143.234.244 - - [18/Mar/2012:13:55:36 -0700] \"GET /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; Trident/4.0)\""} {"log":"66.216.63.42 - Lucille [18/Mar/2012:15:07:29 -0700] \"DELETE /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_3; like Mac OS X) AppleWebKit/533.11 (KHTML, like Gecko) Chrome/52.0.3189.332 Mobile Safari/536.3\""} {"log":"70.137.97.47 - - [18/Mar/2012:19:21:04 -0700] \"GET /lease.pdf HTTP/1.0\" 404 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; Trident/4.0)\""} {"log":"7.160.29.68 - Michael [19/Mar/2012:01:38:17 -0700] \"POST /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (U; Linux i585 x86_64; en-US) Gecko/20130401 Firefox/60.7\""} {"log":"31.119.193.169 - - [19/Mar/2012:08:33:29 -0700] \"GET /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (iPad; CPU iPad OS 7_9_0 like Mac OS X) AppleWebKit/602.13 (KHTML, like Gecko) Chrome/50.0.2327.233 Mobile Safari/534.2\""} ``` A processing rule with the **Source key** value `log`, the **Destination key** value `parsed`, the **Regex** value `^(?[^ ]*) [^ ]* (?[^ ]*) \[(?

Send Amazon EventBridge events to Observability Platform.

Send Buildkite events to Observability Platform.

Send CircleCI events to Observability Platform.

Send GitHub events to Observability Platform.

Send GitLab events to Observability Platform.

Send Google Cloud Personalized Service Health events to Observability Platform.

Send Harness events to Observability Platform.

Send LaunchDarkly events to Observability Platform.

Send Statsig events to Observability Platform.

 # Send Amazon EventBridge events to Observability Platform Source: https://docs.chronosphere.io/ingest/third-party/aws-eventbridge Send Amazon EventBridge events, including AWS Health events, to Chronosphere Observability Platform. Amazon EventBridge is a serverless service you can use to route a stream of real-time data from your applications, third-party software, and Amazon Web Services (AWS) to targets that include HTTP APIs and third-party integrations. You configure routing rules to route events to a target. Amazon EventBridge API destinations are HTTP endpoints you can configure as the target for a rule. Use API destinations to send events from Amazon EventBridge to Chronosphere Observability Platform. You can also use Amazon EventBridge to detect and react to AWS Health events, and send those events to Observability Platform. This configuration is similar to sending [Google Cloud Personalized Service Health events to Observability Platform](/ingest/third-party/google-cloud). To send Amazon EventBridge events to Observability Platform: 1. [Create an API token](#create-an-api-token). 2. [Create an API destination](#create-an-api-destination). 3. Create an Amazon EventBridge rule, depending on which events you want to send to Observability Platform: * [Create an EventBridge rule](#create-an-amazon-eventbridge-rule) for specific events, such as Amazon CloudWatch events. * [Create an EventBridge rule for AWS Health](#create-an-amazon-eventbridge-rule-for-aws-health) events to send those events to Observability Platform. ## Create an API token Before sending events from , contact [Chronosphere Support](/support) to obtain an API token to authenticate with Observability Platform. This API token differs from an API token that's generated when you create a [service account](/administer/accounts-teams/service-accounts). ## Create an API destination To configure Amazon EventBridge to send events to Observability Platform: 1. Create an API destination by completing the steps in the [Amazon EventBridge documentation](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-api-destinations.html#eb-api-destination-create). 2. Configure the API destination you created: 1. For the **Authorization type**, select **API Key**. Enter **API-Token** as your API key name, and enter the API token you received as the value. 2. For the destination endpoint, enter `https://TENANT/api/v1/data/events` and select `POST` as the HTTP method. 3. For the **Invocation rate limit per second**, enter **5** as the value. 4. Select **Use an existing connection** and select the connection you created. ## Create an Amazon EventBridge rule You can create an Amazon EventBridge rule to get notified about specific Amazon EventBridge events in Observability Platform. First, [create an API destination](#create-an-api-destination). Then complete the following steps: 1. [Create an Amazon EventBridge rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) that reacts to events: * Define the rule detail: Enter details for the rule and select **Rule with an event pattern** as the rule type. * Build the event pattern: Select **Use pattern form** as the creation method, and define the event pattern. * Define the target: Select **EventBridge API destination** as the target type, choose your existing API destination from the list, and select **Create a new role for this specific resource**. 2. Expand **Additional settings** and configure an [input transformation](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-transform-target-input.html) to handle mapping from an AWS event to an Observability Platform `CreateEvent` payload. This definition sends the transformed event to a target instead of the original event. You configure an input path to define variables, and then configure an input template to match your event payload. 3. Configure the input path to reference parameters in your event and store those values in variables. For example: ```json theme={null} { "time": "$.time", "resources": "$.resources", "detail": "$.detail", "state": "$.detail.state.value", "prevState": "$.detail.previousState.value", "alarmName": "$.detail.alarmName", "description": "$.detail.configuration.description", "detail": "$.detail", "region": "$.region" } ``` 4. Configure the input template to specify the information you want to pass to your target. This template matches your event payload and uses the variables defined in your input path. You must surround all variables with angle brackets `<>`. For example, the following template defines an Amazon CloudWatch input: ```json theme={null} { "event": { "happenedAt": "