Telemetry Pipeline v3

Use the information on this page to create and manage pipelines in Telemetry Pipeline v3.

Requirements

To use Telemetry Pipeline v3, you must meet the following requirements:

You must have a Kubernetes cluster running Kubernetes v1.24 or later.
You must have Helm v3 or later installed.
You must add the OpenTelemetry Helm charts to your cluster.
You must have yq installed.

Create a pipeline

To create a pipeline in Telemetry Pipeline v3, first build a pipeline configuration file, then deploy the pipeline collector.

Build a pipeline configuration file

Pipelines in Telemetry Pipeline v3 support the same source plugins, destination plugins, and processing rules as v2 pipelines. However, v3 pipelines use a new configuration format. Use one of these methods to build a pipeline configuration file.

Web interface
Manual file creation

You can build a pipeline configuration file in the Telemetry Pipeline web interface, then export that file to use in your Kubernetes cluster.

Sign in to the Telemetry Pipeline web interface.
Open the project where you want to create a pipeline. If none of your projects have Telemetry Pipeline v3 enabled, contact Chronosphere Support.
To determine whether a project is using v2 or v3 of Telemetry Pipeline, check the items listed in the sidebar navigation menu in the Telemetry Pipeline web interface. If Core Instances is displayed in the navigation menu, the active project is using v2, not v3.
Go to Pipelines, then click New Pipeline.
Under Pipeline Name, enter a name for your pipeline.
Use the FluentBit Pipeline Builder and Configuration tabs to configure the source plugins, destination plugins, and processing rules for your pipeline.
After you finish building your configuration file, click Copy Config to copy those configuration settings to your clipboard, or click Download Config to download a YAML configuration file.

You can build a pipeline configuration file by manually creating a YAML file that contains your configuration settings. This file must use a .yaml file extension, not a .yml file extension.

Configuration format for v3 pipelines

The configuration format for v3 pipelines is a superset of the configuration format for v2 pipelines. You can convert a v2 configuration file into a v3 configuration file by inserting the pipeline section from your v2 file into the corresponding section of a v3 file.

Configuration file template

You can replace the highlighted lines with the contents of a v2 configuration file. If you insert the pipeline section from a v2 file into a v3 file, make sure to appropriately indent the inserted content within its parent object.

pipeline-config.yaml

receivers:
  fluentbit:
    fb_config:
      pipeline:
        inputs: []
        outputs: []
processors:
  # configuration processors here
exporters:
  nop: {}
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  chronosphere:
    pipeline:
      pipeline_name: "NAME"
      project_id: "ID"
      project_token: "TOKEN"
      endpoint: "https://core.calyptia.com"
service:
  extensions:
    - health_check
    - chronosphere
  telemetry:
    resource:
      pipeline_name: "NAME"
      project_id: "ID"
    prometheus_enabled: true
    prometheus_port: 8888
  pipelines:
    logs:
      receivers:
        - fluentbit
      processors:
        - # list any active processors
      exporters:
        - nop

Deploy the pipeline collector

After you’ve built a pipeline configuration file, follow these steps to deploy the pipeline collector.

Run the following command to deploy an OpenTelemetry Collector:
```
helm upgrade --install otel-pipeline open-telemetry/opentelemetry-collector \
--set mode=deployment \
  --set image.repository=us-docker.pkg.dev/chronosphereio/otel-pipeline \
  --set image.tag=VERSION \
  -f <(yq '{"alternateConfig": .}' FILE) \
  --namespace NAMESPACE
```
Replace the following values:
- VERSION: The version of the pipeline collector to use.
- FILE: The configuration file to apply to your pipeline.
- NAMESPACE: The namespace where you want to deploy the pipeline collector. To deploy the pipeline collector in a new namespace, include the --create-namespace flag.
To deploy multiple pipelines in the same Kubernetes cluster, use a separate namespace for each pipeline.
Run the following command to verify that the pipeline collector was deployed:
```
kubectl get pods -n NAMESPACE
```
Replace NAMESPACE with the namespace where you deployed the pipeline collector.

Update a pipeline

To update a pipeline, use the same helm upgrade otel-pipeline command that you used to deploy the pipeline collector, but replace the corresponding values of any attributes you want to update.

Upgrade the pipeline collector

To upgrade the pipeline collector, replace VERSION with the latest version of the pipeline collector.

Update pipeline configuration

To update a pipeline’s configuration, replace FILE with the name of your updated pipeline configuration file.

Uninstall a pipeline

To uninstall a pipeline from your cluster, follow these steps:

Run the following Helm command:
```
helm uninstall otel-pipeline --namespace NAMESPACE
```
Replace NAMESPACE with the namespace where you deployed the pipeline collector.
In Pipeline CLI, run the following command:
```
calyptia delete pipeline NAME
```
Replace NAME with the name or UUID of the pipeline to delete.

Additional settings

Use the information in this section to configure additional settings for v3 pipelines in Telemetry Pipeline.

Configure autoscaling

Telemetry Pipeline v3 supports horizontal pod autoscaling (HPA). To enable HPA, include the following flags when you run the helm upgrade otel-pipeline command to deploy or update a pipeline:

  --set autoscaling.enabled=true \
  --set autoscaling.minReplicas=1 \
  --set autoscaling.maxReplicas=10 \
  --set autoscaling.targetCPUUtilizationPercentage=80 \
  --set autoscaling.targetMemoryUtilizationPercentage=70

You can replace the example values with the values of your choosing.

Enable health checks

To enable health checks, configure the OpenTelemetry health_check extension in your pipeline’s configuration file. The OpenTelemetry Helm chart uses this extension for its liveness and readiness probes. For a standard v3 pipeline that uses the recommended OpenTelemetry Helm chart settings, set the value of endpoint to 0.0.0.0:13133:

pipeline-config.yaml

[...]
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
[...]

The configuration builder in the Telemetry Pipeline web interface includes this value by default for new configuration files.

Use Kubernetes metadata for pipeline and agent names

By default, pipeline names are derived from a literal value set in pipeline configuration files, and the agents associated with each pipeline receive automatically generated names. You can override this behavior by using environment variables to assign pipeline and agent names derived from Kubernetes metadata.

When you run the helm upgrade otel-pipeline command to deploy or update a pipeline, include the following flags:
```
  --set "extraEnvs[0].name=POD_NAME" \
  --set "extraEnvs[0].valueFrom.fieldRef.fieldPath=metadata.name" \
  --set "extraEnvs[1].name=DEPLOYMENT_NAME" \
  --set "extraEnvs[1].valueFrom.fieldRef.fieldPath=metadata.labels['app.kubernetes.io/instance']"
```
This creates two environment variables:
- POD_NAME is derived from metadata.name. Because pipelines assign one agent to each Pod, using Pod metadata for agent names guarantees that each agent has a unique name.
- DEPLOYMENT_NAME is derived from metadata.labels['app.kubernetes.io/instance'].
In the extensions.chronosphere.pipeline section of your pipeline’s configuration file, update pipeline_name and agent_name to use the following values:
pipeline-config.yaml
```
[...]
extensions:
 [...]
 chronosphere:
   pipeline:
     pipeline_name: "${env:DEPLOYMENT_NAME}"
     agent_name: "${env:POD_NAME}"
[...]
```
Agent names must be unique within a pipeline. Never assign a value to agent_name that would attempt to assign the same name to multiple agents.

Route data from Fluent Bit to OpenTelemetry

In both Telemetry Pipeline v2 and v3, pipeline plugins and processing rules are built using Fluent Bit. In Telemetry Pipeline v3, that Fluent Bit configuration is wrapped in an OpenTelemetry Collector. You can route data from the underlying Fluent Bit pipeline to the OpenTelemetry Collector for further processing.

In the receivers.fluentbit.fb_config.pipeline.outputs section of your pipeline’s configuration file, add an http output with the following settings:

pipeline-config.yaml

receivers:
  fluentbit:
    fb_config:
      pipeline:
        [...]
        outputs:
        - name: http
          match: '*'
          format: msgpack
          port: $${FORWARD_HTTP_PORT} # this is a literal value, not an environment variable
[...]

In the processors section of your pipeline’s configuration file, add the memory_limiter and batch processors.
The memory_limiter processor must execute before any other OpenTelemetry processors, which means it must always be the first processor in the processors list.
pipeline-config.yaml
```
[...]
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 90
    spike_limit_percentage: 10
  batch: {}
  [...]
```

Add memory_limiter and batch to the service.pipelines.logs.processors section of your pipeline’s configuration file:

pipeline-config.yaml

[...]
service:
 [...]
  pipelines:
    logs:
      receivers:
        - fluentbit
      processors:
        - memory_limiter
        - batch
      [...]

Use the helm upgrade otel-pipeline to deploy or update your pipeline, and add the following flags:
```
  --set resources.requests.cpu=500m \
  --set resources.requests.memory=512Mi \
  --set resources.limits.cpu=1 \
  --set resources.limits.memory=1Gi
```
These values set the resource limits for the memory_limiter processor. You can replace the example values with the values of your choosing.

After you route data from Fluent Bit to OpenTelemetry, you can configure other OpenTelemetry receivers, processors, and exporters in the service section of your pipeline’s configuration file. It’s also possible to use standard Fluent Bit-based plugins alongside the http plugin that routes data to OpenTelemetry. However, any data routed through standard Fluent Bit-based plugins won’t be affected by OpenTelemetry processors and exporters.

Internal telemetry

You can configure internal telemetry for v3 pipelines in the service.telemetry section of pipeline configuration files.

Example configuration

pipeline-config.yaml

[...]
service:
  [...]
  telemetry:
    resource:
      custom_key: value
    control_plane_metrics_enabled: true
    control_plane_metrics_interval: 60s
    control_plane_metrics_timeout: 30s
    prometheus_enabled: true
    prometheus_port: 8888
    logs:
      # configure standard OpenTelemetry Collector settings for logs here
    traces:
      # configure standard OpenTelemetry Collector settings for traces here
[...]

Logs and traces

Telemetry Pipeline v3 uses the standard OpenTelemetry Collector configuration settings for internal logs and traces.

Metrics

Telemetry Pipeline v3 overrides the standard OpenTelemetry Collector configuration settings for internal metrics. Instead, metrics use the following configuration settings:

Key	Description	Default
`control_plane_metrics_enabled`	Whether to send internal metrics to the Telemetry Pipeline backend.	`true`
`control_plane_metrics_interval`	The interval for sending metrics to the Telemetry Pipeline backend.	`60s`
`control_plane_metrics_timeout`	The timeout for sending metrics to the Telemetry Pipeline backend.	`30s`
`prometheus_enabled`	Whether to enable the local Prometheus endpoint.	`true`
`prometheus_port`	The port for the Prometheus endpoint where metrics are available.	`8888`

Resource attributes

Telemetry Pipeline v3 adds a custom internal telemetry type for resource attributes. These resource attributes are used to enrich other internal telemetry data, which means the values of these resource attributes will affect the internal logs, traces, and metrics emitted by your pipeline. You can add custom key/value pairs in the service.telemetry.resource section of your pipeline’s configuration file to enrich your pipeline’s internal telemetry with custom resource attributes. These keys and value must be strings, and key names must not overlap with the names of the keys that Telemetry Pipeline automatically assigns to pipelines.

Any key/value pairs included in service.telemetry.resource are sent to the Telemetry Pipeline backend to enrich internal telemetry. Don’t include sensitive data in these key/value pairs.

In addition to any custom resource attributes you specify, Telemetry Pipeline assigns these resource attributes to all pipelines. Although it’s possible to override the default values for these resource attributes, Chronosphere doesn’t recommend doing so.

Key	Description	Default value
`pipeline_name`	The name of your pipeline.	The equivalent value specified in `service.extensions.chronosphere.pipeline`
`agent_name`	The names of the agents associated with your pipeline. Agent names must be unique within a pipeline. Never assign a value to `agent_name` that would attempt to assign the same name to multiple agents.	The equivalent value specified in `service.extensions.chronosphere.pipeline`
`project_id`	The ID of the Telemetry Pipeline project to which your pipeline belongs.	The equivalent value specified in `service.extensions.chronosphere.pipeline`
`host.name`	Your host name.	Auto-detected value of your host name, if possible
`service.name`	The name of your pipeline collector.	`chrono-telemetry-pipeline`
`service.version`	The version of your pipeline collector.	The current version of your pipeline collector

Supported Pipeline CLI commands

Telemetry Pipeline v3 supports the following Pipeline CLI commands.

The steps to install and authenticate in Pipeline CLI are unchanged in Telemetry Pipeline v3.

Command list

`get pipelines`

Returns a list of deployed pipelines.

calyptia get pipelines

`get pipeline`

Returns information about a pipeline.

Add the flag --only-config to return only the contents of that pipeline’s configuration file.

calyptia get pipeline NAME

Replace NAME with the name or UUID of your pipeline.

`get agents`

Returns a list of agents connected to a pipeline.

calyptia get agents --pipeline NAME

Replace NAME with the name or UUID of your pipeline.

`get agent`

Returns information about a specified agent.

copy

calyptia get agent UUID

Replace UUID with the UUID of an agent.

`get project`

Returns the name of the active Pipeline CLI project.

calyptia get project

`delete pipelines`

Removes all pipelines associated with the active project from the list of pipelines in the Telemetry Pipeline web interface.

If you run this command without using Helm to uninstall the pipelines from your cluster first, any active pipelines will reappear in the Telemetry Pipeline web interface. However, this command permanently removes any inactive pipelines from the Telemetry Pipeline web interface.

calyptia delete pipelines

`delete pipeline`

Removes the specified pipeline from the list of pipelines in the Telemetry Pipeline web interface.

If you run this command for an active pipeline without uninstalling the pipeline from your cluster from your cluster first, the pipeline will reappear in the Telemetry Pipeline web interface. However, this command permanently removes any inactive pipelines from the Telemetry Pipeline web interface.

calyptia delete pipeline NAME

Replace NAME with the name or UUID of the pipeline to delete.

`version`

Returns the currently installed version of Pipeline CLI.

calyptia version

Observability Platform

Telemetry Pipeline

More information

Requirements

Create a pipeline

Build a pipeline configuration file

Configuration format for v3 pipelines

Deploy the pipeline collector

Update a pipeline

Upgrade the pipeline collector

Update pipeline configuration

Uninstall a pipeline

Additional settings

Configure autoscaling

Enable health checks

Use Kubernetes metadata for pipeline and agent names

Route data from Fluent Bit to OpenTelemetry

Internal telemetry

Logs and traces

Metrics

Resource attributes

Supported Pipeline CLI commands

`get pipelines`

`get pipeline`

`get agents`

`get agent`

`get project`

`delete pipelines`

`delete pipeline`

`version`

Observability Platform

Telemetry Pipeline

More information

Documentation Index

​Requirements

​Create a pipeline

​Build a pipeline configuration file

​Configuration format for v3 pipelines

​Deploy the pipeline collector

​Update a pipeline

​Upgrade the pipeline collector

​Update pipeline configuration

​Uninstall a pipeline

​Additional settings

​Configure autoscaling

​Enable health checks

​Use Kubernetes metadata for pipeline and agent names

​Route data from Fluent Bit to OpenTelemetry

​Internal telemetry

​Logs and traces

​Metrics

​Resource attributes

​Supported Pipeline CLI commands

​get pipelines

​get pipeline

​get agents

​get agent

​get project

​delete pipelines

​delete pipeline

​version

Requirements

Create a pipeline

Build a pipeline configuration file

Configuration format for v3 pipelines

Deploy the pipeline collector

Update a pipeline

Upgrade the pipeline collector

Update pipeline configuration

Uninstall a pipeline

Additional settings

Configure autoscaling

Enable health checks

Use Kubernetes metadata for pipeline and agent names

Route data from Fluent Bit to OpenTelemetry

Internal telemetry

Logs and traces

Metrics

Resource attributes

Supported Pipeline CLI commands

`get pipelines`

`get pipeline`

`get agents`

`get agent`

`get project`

`delete pipelines`

`delete pipeline`

`version`