Pipeline operations

You can use Pipeline CLI to perform the following pipeline operations.

Get pipeline IDs

In addition to a name, Telemetry Pipeline assigns each pipeline a unique numerical ID.

To retrieve a list of pipeline IDs, run the following command:

calyptia get pipelines --core-instance INSTANCE --show-ids

Replace INSTANCE with the name of your Core Instance.

View configuration history

Telemetry Pipeline saves a history of each pipeline's configuration. To view this history, you can run either of the following commands:

calyptia get pipeline_config_history --pipeline PIPELINE

calyptia get pipeline PIPELINE --include-config-history

Replace PIPELINE with either the name or unique ID of the pipeline whose history you'd like to view.

Limit results

Use the --last flag to display only a subset of snapshots. For example, to return a list of the last five snapshots, run:

calyptia get pipeline_config_history --pipeline my-pipeline --last 5

View individual changes

Use the -o yaml flag to view the individual changes behind each snapshot:

calyptia get pipeline_config_history --pipeline my-pipeline -o yaml

Revert pipeline configuration

To revert a pipeline to a previous configuration, use the rollout command:

calyptia rollout pipeline PIPELINE --to-config-id HISTORY

Replace PIPELINE with either the name or unique ID of the pipeline whose configuration you'd like to revert and HISTORY with the relevant snapshot ID.

You can also use the --steps-back flag to revert a pipeline by specifying how many steps to move back. For example, to revert a pipeline named my-pipeline to its immediate previous configuration, run:

calyptia rollout pipeline my-pipeline --steps-back 1

Upgrade pipeline version

To upgrade a pipeline's underlying binary file, use the update pipeline command and the --image flag:

calyptia update pipeline PIPELINE --image ghcr.io/calyptia/core/calyptia-fluent-bit:VERSION

Replace PIPELINE with either the name or unique ID of the pipeline you'd like to modify and VERSION with the latest version of the pipeline.

Modify pipeline network settings

After you deploy or update a pipeline that has a network-based source, Telemetry Pipeline automatically load-balances that pipeline using Kubernetes services (even if you deployed Telemetry Pipeline on a Linux server).

By default, Telemetry Pipeline uses the LoadBalancer network service. You can modify this behavior by running the following command:

calyptia update pipeline PIPELINE --service-type SERVICE

Replace PIPELINE with either the name or unique ID of the pipeline whose network setting you'd like to modify and SERVICE with the service type you'd like to use. Possible service type values are LoadBalancer, NodePort, and ClusterIP.

Add pipeline files

By default, each pipeline includes a file that lists the parsers used by that pipeline. You can also add your own files to a pipeline, including certificates, stream processor jobs, and custom Lua scripts.

To add files to a pipeline, use the --file flag:

calyptia update pipeline PIPELINE --file FILENAME

Replace PIPELINE with either the name or unique ID of your pipeline and FILENAME with the name of the file to add. Files must be less than 10 MB and must not include a file extension.

After you've added a file, you can access it through the {{ files.FILENAME }} variable in configuration files and the Telemetry Pipeline web interface.

Add encrypted file

To encrypt pipeline files, use the --encrypt-files flag. Telemetry Pipeline uses RSA‑OAEP encryption with a 2048-bit key.