> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Telemetry Pipeline v3

> Create and manage v3 pipelines in Chronosphere Telemetry Pipeline.

Use the information on this page to create and manage pipelines in Telemetry Pipeline v3.

## Requirements

To use Telemetry Pipeline v3, you must meet the following requirements:

* You must have a Kubernetes cluster running Kubernetes v1.24 or later.
* You must have Helm v3 or later installed.
* You must add the
  [OpenTelemetry Helm charts](https://open-telemetry.github.io/opentelemetry-helm-charts)
  to your cluster.
* You must have [yq](https://github.com/mikefarah/yq) installed.

## Create a pipeline

To create a pipeline in Telemetry Pipeline v3, first
[build a pipeline configuration file](#build-a-pipeline-configuration-file), then
[deploy the pipeline collector](#deploy-the-pipeline-collector).

### Build a pipeline configuration file

Pipelines in Telemetry Pipeline v3 support the same
[source plugins](/ingest/pipeline/plugins/source-plugins),
[destination plugins](/ingest/pipeline/plugins/destination-plugins), and
[processing rules](/ingest/pipeline/processing-rules) as v2 pipelines. However,
v3 pipelines use a new [configuration format](#configuration-format-for-v3-pipelines).

Use one of these methods to build a pipeline configuration file.

<Tabs>
  <Tab title="Web interface" id="build-config-web">
    You can build a pipeline configuration file in the Telemetry Pipeline web interface,
    then export that file to use in your Kubernetes cluster.

    1. Sign in to the [Telemetry Pipeline web interface](https://core.calyptia.com/).

    2. Open the project where you want to create a pipeline. If none of your projects
       have Telemetry Pipeline v3 enabled, contact [Chronosphere Support](/support).

           <Note>
             To determine whether a project is using v2 or v3 of Telemetry Pipeline,
             check the items listed in the sidebar navigation menu in the Telemetry Pipeline
             web interface. If **Core Instances** is displayed in the navigation menu,
             the active project is using v2, not v3.
           </Note>

    3. Go to **Pipelines**, then click **New Pipeline**.

    4. Under **Pipeline Name**, enter a name for your pipeline.

    5. Use the **FluentBit Pipeline Builder** and **Configuration** tabs to configure
       the source plugins, destination plugins, and processing rules for your pipeline.

    6. After you finish building your configuration file, click **Copy Config** to
       copy those configuration settings to your clipboard, or click **Download Config**
       to download a YAML configuration file.
  </Tab>

  <Tab title="Manual file creation" id="build-config-manual">
    You can build a pipeline configuration file by manually creating a YAML file that
    contains your configuration settings. This file must use a `.yaml` file extension,
    not a `.yml` file extension.
  </Tab>
</Tabs>

#### Configuration format for v3 pipelines

The configuration format for v3 pipelines is a superset of the configuration
format for v2 pipelines. You can convert a v2 configuration file into a v3 configuration
file by inserting the `pipeline` section from your v2 file into the corresponding
section of a v3 file.

<Accordion title="Configuration file template">
  You can replace the highlighted lines with the contents of a v2 configuration file.
  If you insert the `pipeline` section from a v2 file into a v3 file, make sure to
  appropriately indent the inserted content within its parent object.

  ```yaml title="pipeline-config.yaml" highlight={4-6} theme={null}
  receivers:
    fluentbit:
      fb_config:
        pipeline:
          inputs: []
          outputs: []
  processors:
    # configuration processors here
  exporters:
    nop: {}
  extensions:
    health_check:
      endpoint: 0.0.0.0:13133
    chronosphere:
      pipeline:
        pipeline_name: "NAME"
        project_id: "ID"
        project_token: "TOKEN"
        endpoint: "https://core.calyptia.com"
  service:
    extensions:
      - health_check
      - chronosphere
    telemetry:
      resource:
        pipeline_name: "NAME"
        project_id: "ID"
      prometheus_enabled: true
      prometheus_port: 8888
    pipelines:
      logs:
        receivers:
          - fluentbit
        processors:
          - # list any active processors
        exporters:
          - nop
  ```
</Accordion>

### Deploy the pipeline collector

After you've [built a pipeline configuration file](#build-a-pipeline-configuration-file),
follow these steps to deploy the pipeline collector.

1. Run the following command to deploy an [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector):

   ```shell theme={null}
   helm upgrade --install otel-pipeline open-telemetry/opentelemetry-collector \
   --set mode=deployment \
     --set image.repository=us-docker.pkg.dev/chronosphereio/otel-pipeline/otel-pipeline \
     --set image.tag=VERSION \
     -f <(yq '{"alternateConfig": .}' FILE) \
     --namespace NAMESPACE
   ```

   Replace the following values:

   * *`VERSION`*: The version of the pipeline collector to use (for example,
     `v3.0.0`).
   * *`FILE`*: The [configuration file](#build-a-pipeline-configuration-file)
     to apply to your pipeline.
   * *`NAMESPACE`*: The namespace where you want to deploy the pipeline collector.
     To deploy the pipeline collector in a new namespace, append the `--create-namespace`
     flag after `--namespace NAMESPACE`.

   <Note>
     To deploy multiple pipelines in the same Kubernetes cluster, use a separate namespace
     for each pipeline.
   </Note>

2. Run the following command to verify that the pipeline collector was deployed:

   ```shell theme={null}
   kubectl get pods -n NAMESPACE
   ```

   Replace *`NAMESPACE`* with the namespace where you deployed the pipeline collector.

## Update a pipeline

To update a pipeline, use the same `helm upgrade otel-pipeline` command that
you used to [deploy the pipeline collector](#deploy-the-pipeline-collector), but replace
the corresponding values of any attributes you want to update.

### Upgrade the pipeline collector

To upgrade the pipeline collector, replace `VERSION` with the latest version of the
pipeline collector.

### Update pipeline configuration

To update a pipeline's configuration, replace `FILE` with the name of your updated
pipeline configuration file.

## Uninstall a pipeline

To uninstall a pipeline from your cluster, follow these steps:

1. Run the following Helm command:

   ```shell theme={null}
   helm uninstall otel-pipeline --namespace NAMESPACE
   ```

   Replace *`NAMESPACE`* with the namespace where you deployed the pipeline collector.

2. In Pipeline CLI, run the following command:

   ```shell theme={null}
   calyptia delete pipeline NAME
   ```

   Replace *`NAME`* with the name or UUID of the pipeline to delete.

## Additional settings

Use the information in this section to configure additional settings for
v3 pipelines in Telemetry Pipeline.

### Configure autoscaling

Telemetry Pipeline v3 supports
[horizontal pod autoscaling (HPA)](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/).
To enable HPA, include the following flags when you run the `helm upgrade otel-pipeline`
command to [deploy](#deploy-the-pipeline-collector) or [update](#update-a-pipeline)
a pipeline:

```shell theme={null}
  --set autoscaling.enabled=true \
  --set autoscaling.minReplicas=1 \
  --set autoscaling.maxReplicas=10 \
  --set autoscaling.targetCPUUtilizationPercentage=80 \
  --set autoscaling.targetMemoryUtilizationPercentage=70
```

You can replace the example values with the values of your choosing.

### Enable health checks

To enable health checks, configure the OpenTelemetry `health_check` extension
in your pipeline's [configuration file](#configuration-format-for-v3-pipelines).
The OpenTelemetry Helm chart uses this extension for its liveness and readiness
probes.

For a standard v3 pipeline that uses the recommended OpenTelemetry Helm chart
settings, set the value of `endpoint` to `0.0.0.0:13133`:

```yaml title="pipeline-config.yaml" theme={null}
[...]
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
[...]
```

The [configuration builder](#build-a-pipeline-configuration-file) in the Telemetry
Pipeline web interface includes this value by default for new configuration files.

### Use Kubernetes metadata for pipeline and agent names

By default, pipeline names are derived from a literal value set in pipeline
configuration files, and the agents associated with each pipeline receive
automatically generated names. You can override this behavior by using environment
variables to assign pipeline and agent names derived from Kubernetes metadata.

1. When you run the `helm upgrade otel-pipeline` command to
   [deploy](#deploy-the-pipeline-collector) or [update](#update-a-pipeline)
   a pipeline, include the following flags:

   ```shell theme={null}
     --set "extraEnvs[0].name=POD_NAME" \
     --set "extraEnvs[0].valueFrom.fieldRef.fieldPath=metadata.name" \
     --set "extraEnvs[1].name=DEPLOYMENT_NAME" \
     --set "extraEnvs[1].valueFrom.fieldRef.fieldPath=metadata.labels['app.kubernetes.io/instance']"
   ```

   This creates two environment variables:

   * `POD_NAME` is derived from `metadata.name`. Because pipelines assign one agent
     to each Pod, using Pod metadata for agent names guarantees that each agent
     has a unique name.
   * `DEPLOYMENT_NAME` is derived from `metadata.labels['app.kubernetes.io/instance']`.

2. In the `extensions.chronosphere.pipeline` section of your pipeline's configuration
   file, update `pipeline_name` and `agent_name` to use the following values:

   ```yaml title="pipeline-config.yaml" theme={null}
   [...]
   extensions:
    [...]
    chronosphere:
      pipeline:
        pipeline_name: "${env:DEPLOYMENT_NAME}"
        agent_name: "${env:POD_NAME}"
   [...]
   ```

   <Note>
     Agent names must be unique within a pipeline. Never assign a value to `agent_name`
     that would attempt to assign the same name to multiple agents.
   </Note>

### Route data from Fluent Bit to OpenTelemetry

In both Telemetry Pipeline v2 and v3, pipeline plugins and processing rules
are built using [Fluent Bit](https://fluentbit.io/). In Telemetry Pipeline v3, that Fluent Bit configuration
is wrapped in an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/).
You can route data from the underlying Fluent Bit pipeline to the OpenTelemetry
Collector for further processing.

1. In the `receivers.fluentbit.fb_config.pipeline.outputs` section of your pipeline's
   configuration file, add an `http` output with the following settings:

   ```yaml title="pipeline-config.yaml" theme={null}
   receivers:
     fluentbit:
       fb_config:
         pipeline:
           [...]
           outputs:
           - name: http
             match: '*'
             format: msgpack
             port: $${FORWARD_HTTP_PORT} # this is a literal value, not an environment variable
   [...]
   ```

2. In the `processors` section of your pipeline's configuration file, add the
   `memory_limiter` and `batch` processors.

   <Note>
     The `memory_limiter` processor must execute before any other OpenTelemetry
     processors, which means it must always be the first processor in the `processors` list.
   </Note>

   ```yaml title="pipeline-config.yaml" theme={null}
   [...]
   processors:
     memory_limiter:
       check_interval: 1s
       limit_percentage: 90
       spike_limit_percentage: 10
     batch: {}
     [...]
   ```

3. Add `memory_limiter` and `batch` to the `service.pipelines.logs.processors`
   section of your pipeline's configuration file:

   ```yaml title="pipeline-config.yaml" theme={null}
   [...]
   service:
    [...]
     pipelines:
       logs:
         receivers:
           - fluentbit
         processors:
           - memory_limiter
           - batch
         [...]
   ```

4. Use the `helm upgrade otel-pipeline` to [deploy](#deploy-the-pipeline-collector)
   or [update](#update-a-pipeline) your pipeline, and add the following flags:

   ```shell theme={null}
     --set resources.requests.cpu=500m \
     --set resources.requests.memory=512Mi \
     --set resources.limits.cpu=1 \
     --set resources.limits.memory=1Gi
   ```

   These values set the resource limits for the `memory_limiter` processor. You can
   replace the example values with the values of your choosing.

After you route data from Fluent Bit to OpenTelemetry, you can configure other
OpenTelemetry receivers, processors, and exporters in the `service` section of
your pipeline's configuration file. It's also possible to use standard Fluent
Bit-based plugins alongside the `http` plugin that routes data to OpenTelemetry.
However, any data routed through standard Fluent Bit-based plugins won't be
affected by OpenTelemetry processors and exporters.

### Internal telemetry

You can configure internal telemetry for v3 pipelines in the `service.telemetry`
section of pipeline configuration files.

<Accordion title="Example configuration">
  ```yaml title="pipeline-config.yaml" theme={null}
  [...]
  service:
    [...]
    telemetry:
      resource:
        custom_key: value
      control_plane_metrics_enabled: true
      control_plane_metrics_interval: 60s
      control_plane_metrics_timeout: 30s
      prometheus_enabled: true
      prometheus_port: 8888
      logs:
        # configure standard OpenTelemetry Collector settings for logs here
      traces:
        # configure standard OpenTelemetry Collector settings for traces here
  [...]
  ```
</Accordion>

#### Logs and traces

Telemetry Pipeline v3 uses the standard OpenTelemetry Collector configuration settings for
internal [logs](https://opentelemetry.io/docs/collector/internal-telemetry/#configure-internal-logs)
and [traces](https://opentelemetry.io/docs/collector/internal-telemetry/#configure-internal-traces).

#### Metrics

Telemetry Pipeline v3 overrides the standard OpenTelemetry Collector configuration
settings for internal metrics. Instead, metrics use the following configuration settings:

| Key                              | Description                                                         | Default |
| -------------------------------- | ------------------------------------------------------------------- | ------- |
| `control_plane_metrics_enabled`  | Whether to send internal metrics to the Telemetry Pipeline backend. | `true`  |
| `control_plane_metrics_interval` | The interval for sending metrics to the Telemetry Pipeline backend. | `60s`   |
| `control_plane_metrics_timeout`  | The timeout for sending metrics to the Telemetry Pipeline backend.  | `30s`   |
| `prometheus_enabled`             | Whether to enable the local Prometheus endpoint.                    | `true`  |
| `prometheus_port`                | The port for the Prometheus endpoint where metrics are available.   | `8888`  |

#### Resource attributes

Telemetry Pipeline v3 adds a custom internal telemetry type for resource attributes.
These resource attributes are used to enrich other internal telemetry data, which
means the values of these resource attributes will affect the internal logs, traces, and
metrics emitted by your pipeline.

You can add custom key/value pairs in the `service.telemetry.resource` section of
your pipeline's configuration file to enrich your pipeline's internal telemetry with
custom resource attributes. These keys and value must be strings, and key names must
not overlap with the names of the keys that Telemetry Pipeline automatically assigns
to pipelines.

<Note>
  Any key/value pairs included in `service.telemetry.resource` are sent to the Telemetry
  Pipeline backend to enrich internal telemetry. Don't include sensitive data in
  these key/value pairs.
</Note>

In addition to any custom resource attributes you specify, Telemetry Pipeline assigns
these resource attributes to all pipelines. Although it's possible to override the
default values for these resource attributes, Chronosphere doesn't recommend doing
so.

| Key               | Description                                                                                                                                                                                              | Default value                                                                |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| `pipeline_name`   | The name of your pipeline.                                                                                                                                                                               | The equivalent value specified in `service.extensions.chronosphere.pipeline` |
| `agent_name`      | The names of the agents associated with your pipeline. Agent names must be unique within a pipeline. Never assign a value to `agent_name` that would attempt to assign the same name to multiple agents. | The equivalent value specified in `service.extensions.chronosphere.pipeline` |
| `project_id`      | The ID of the Telemetry Pipeline project to which your pipeline belongs.                                                                                                                                 | The equivalent value specified in `service.extensions.chronosphere.pipeline` |
| `host.name`       | Your host name.                                                                                                                                                                                          | Auto-detected value of your host name, if possible                           |
| `service.name`    | The name of your pipeline collector.                                                                                                                                                                     | `chrono-telemetry-pipeline`                                                  |
| `service.version` | The version of your pipeline collector.                                                                                                                                                                  | The current version of your pipeline collector                               |

## Supported Pipeline CLI commands

Telemetry Pipeline v3 supports the following Pipeline CLI commands.

<Note>
  The steps to [install](/ingest/pipeline/pipeline-cli/install) and
  [authenticate](/ingest/pipeline/pipeline-cli/authenticate) in Pipeline CLI are unchanged in
  Telemetry Pipeline v3.
</Note>

<Accordion title="Command list">
  ### `get pipelines`

  Returns a list of deployed pipelines.

  ```shell theme={null}
  calyptia get pipelines
  ```

  ### `get pipeline`

  Returns information about a pipeline.

  <Note>
    Add the flag `--only-config` to return only the contents of that pipeline's
    configuration file.
  </Note>

  ```shell theme={null}
  calyptia get pipeline NAME
  ```

  Replace *`NAME`* with the name or UUID of your pipeline.

  ### `get agents`

  Returns a list of agents connected to a pipeline.

  ```shell theme={null}
  calyptia get agents --pipeline NAME
  ```

  Replace *`NAME`* with the name or UUID of your pipeline.

  ### `get agent`

  Returns information about a specified agent.

  ```shell copy theme={null}
  calyptia get agent UUID
  ```

  Replace *`UUID`* with the UUID of an agent.

  ### `get project`

  Returns the name of the active Pipeline CLI project.

  ```shell theme={null}
  calyptia get project
  ```

  ### `delete pipelines`

  Removes all pipelines associated with the active project from the list of pipelines
  in the Telemetry Pipeline web interface.

  <Note>
    If you run this command without using Helm to
    [uninstall the pipelines from your cluster](#uninstall-a-pipeline) first, any active
    pipelines will reappear in the Telemetry Pipeline web interface. However, this command
    permanently removes any inactive pipelines from the Telemetry Pipeline web interface.
  </Note>

  ```shell theme={null}
  calyptia delete pipelines
  ```

  ### `delete pipeline`

  Removes the specified pipeline from the list of pipelines in the Telemetry
  Pipeline web interface.

  <Note>
    If you run this command for an active pipeline without
    [uninstalling the pipeline from your cluster](#uninstall-a-pipeline) from your cluster
    first, the pipeline will reappear in the Telemetry Pipeline web interface.
    However, this command permanently removes any inactive pipelines from the Telemetry
    Pipeline web interface.
  </Note>

  ```shell theme={null}
  calyptia delete pipeline NAME
  ```

  Replace *`NAME`* with the name or UUID of the pipeline to delete.

  ### `version`

  Returns the currently installed version of Pipeline CLI.

  ```shell theme={null}
  calyptia version
  ```
</Accordion>