Create and modify pipelines

You can use Chronosphere Telemetry Pipeline to create, modify, and delete pipelines.

Create a pipeline

Select from the following methods to create a pipeline.

Web interface
Pipeline CLI

To create a pipeline using the Telemetry Pipeline web interface:

Sign in to Telemetry Pipeline.
Open the project where you want to deploy a new pipeline.
Go to Core Instances, and then click the name of the Core Instance running in the environment where you want to deploy a new pipeline.
In the Data Pipelines section, click Create Data Pipeline.
In the dialog that appears, configure the following options:
- Name: The name to give your pipeline.
- Service type: The Service type to use for your pipeline.
- Workload type: The Workload type to use for your pipeline.
- Storage class: (StatefulSet pipelines only) The StorageClass to use for each Pod’s PersistentVolume. After you set this value for a StatefulSet pipeline, you can’t modify it.
- Retention policy when scaled: (StatefulSet pipelines only) Specifies whether Kubernetes should retain or delete the PersistentVolumeClaim (PVC) associated with each affected Pod if the StatefulSet pipeline is scaled down.
- Retention policy when deleted: (StatefulSet pipelines only) Specifies whether Kubernetes should retain or delete each Pod’s PVC if the StatefulSet pipeline is deleted.
Click Save and continue.
Add and configure your source plugins and destination plugins.
Add processing rules to transform the telemetry data as it passes through your pipeline.
Click Save and Deploy to finish creating your pipeline.

To create a pipeline using Pipeline CLI:

Create a pipeline configuration file. For example, this configuration file specifies a pipeline that uses mock data as a source and standard output as a destination:

pipeline:
   inputs:
      - dummy: '{"message":"dummy"}'
        rate: "1"
        samples: "0"
        start_time_sec: "-1"
        start_time_nsec: "-1"
        name: dummy
        tag: dummy.92a99f85-50b8-401d-a619-a52b14288f21
   filters:
      - name: lua
        match: dummy.92a99f85-50b8-401d-a619-a52b14288f21
   outputs:
      - format: json
        Name: stdout
        Match_Regex: ^(?!.*_calyptia_cloud).*$

Run the following command:
```
calyptia create pipeline \
--core-instance INSTANCE \
--name PIPELINE \
--image ghcr.io/chronosphereio/core/calyptia-fluent-bit:VERSION
--config-file CONFIG \
```
Replace the following:
- INSTANCE: The name of your Core Instance.
- PIPELINE: The name to give your new pipeline.
- VERSION: Optional. The latest version of the pipeline agent. You can find this value in the Pipeline agent column of the Telemetry Pipeline component versions table. For example, if the latest version of the pipeline agent is 1.23.0, provide the value 1.23.0 to use the Red Hat UBI agent image or 1.23.0-distroless to use the distroless agent image. If you don’t specify a version number, Telemetry Pipeline uses the latest version of the Red Hat UBI agent image by default.
- CONFIG: Optional. The name of your pipeline configuration file. If you don’t specify a configuration file, Telemetry Pipeline creates an empty placeholder that you can modify at a later time. If you specify a configuration file, you can also add the --skip-config-validation flag to skip the validation process for that file.

Telemetry Pipeline assigns a full name to each pipeline using the [core-instance-name]-[pipeline_name] naming schema, where [core-instance-name] is the name of your Core Instance and [pipeline-name] is the short name you gave your pipeline during the calyptia create process.

Modify a pipeline

Select from the following methods to modify a pipeline.

To upgrade a pipeline to the latest version, you must use Pipeline CLI. For more information, see Upgrade a pipeline.

Web interface
Pipeline CLI

To modify a pipeline in the Telemetry Pipeline web interface:

Sign in to Telemetry Pipeline.
Open the project that contains the pipeline that you want to modify.
Go to Core Instances, then click the name of the Core Instance associated with the pipeline you want to modify.
Under Data Pipelines, click the name of the pipeline that you want to modify.
Click Edit.
Configure the source, destination, and processing rules for your pipeline.
Click Save and Deploy to save your changes.
To manage advanced pipeline settings, go to the Advanced Settings page. The settings on this page include:
- Replicas
- Deployment strategy
- Version and agent image type
- Files
- Secrets

Delete a pipeline

Select from the following methods to delete a pipeline.

Web interface
Pipeline CLI

To delete a pipeline in the Telemetry Pipeline web interface:

Sign in to Telemetry Pipeline.
Open the project that contains the pipeline that you want to delete.
Go to Core Instances, then click the name of the Core Instance associated with the pipeline you want to delete.
Under Data Pipelines, find the pipeline that you want to delete, and then click Settings > Delete.
In the confirmation dialog that appears, click Delete to permanently delete your pipeline.

To delete a pipeline using Pipeline CLI:

Run the following command:
```
calyptia delete pipeline PIPELINE
```
Replace PIPELINE with either the name or unique ID of the pipeline to delete.
Optional: If you installed Telemetry Pipeline in a Kubernetes cluster, use kubectl to delete the CRDs associated with your pipeline. This ensures no stray pipeline resources persist in your cluster. To do so, run the following command:
```
kubectl delete pipeline NAME
```
Replace NAME with the name of the pipeline you deleted in the previous step.

Observability Platform

Telemetry Pipeline

Tools

More information

Create a pipeline

Modify a pipeline

Delete a pipeline

Observability Platform

Telemetry Pipeline

Tools

More information

​Create a pipeline

​Modify a pipeline

​Delete a pipeline

Create a pipeline

Modify a pipeline

Delete a pipeline