Pipeline operations

You can use Calyptia CLI to perform the following pipeline operations.

Create a pipeline

To create a pipeline:

1. Create a YAML configuration file. For example, this configuration file specifies a pipeline that uses mock data as a source and standard output as a destination:

   pipeline:
      inputs:
         - dummy: '{"message":"dummy"}'
            rate: "1"
            samples: "0"
            start_time_sec: "-1"
            start_time_nsec: "-1"
            Name: dummy
            tag: dummy.92a99f85-50b8-401d-a619-a52b14288f21
      filters:
         - Name: lua
            match: dummy.92a99f85-50b8-401d-a619-a52b14288f21
            active: "false"
      outputs:
         - format: json
            Name: stdout
            Match_Regex: ^(?!.*_calyptia_cloud).*$

2. Run the following command:

   calyptia create pipeline --config-file CONFIG --core-instance INSTANCE --name PIPELINE --skip-config-validation

   Replace CONFIG with the name of your configuration file, INSTANCE with the name of your Core Instance, and PIPELINE with a short name you'd like to give your new pipeline.

Calyptia assigns a full name to each pipeline using the [core-instance-name]-[pipeline_name] naming schema, where [core-instance-name] is the name of your Core Instance and [pipeline-name] is the short name you gave your pipeline during the calyptia create process.

Get pipeline IDs

In addition to a name, Calyptia assigns each pipeline a unique numerical ID.

To retrieve a list of pipeline IDs, run the following command:

calyptia get pipelines --core-instance INSTANCE --show-ids

Replace INSTANCE with the name of your Core Instance.

View configuration history

Calyptia saves a history of each pipeline's configuration. To view this history, you can run either of the following commands:

calyptia get pipeline_config_history --pipeline PIPELINE

calyptia get pipeline PIPELINE --include-config-history

Replace PIPELINE with either the name or unique ID of the pipeline whose history you'd like to view.

Limit results

Use the --last flag to display only a subset of snapshots. For example, to return a list of the last five snapshots, run:

calyptia get pipeline_config_history --pipeline my-pipeline --last 5

View individual changes

Use the -o yaml flag to view the individual changes behind each snapshot:

calyptia get pipeline_config_history --pipeline my-pipeline -o yaml

Revert pipeline configuration

To revert a pipeline to a previous configuration, use the rollout command:

calyptia rollout pipeline PIPELINE --to-config-id HISTORY

Replace PIPELINE with either the name or unique ID of the pipeline whose configuration you'd like to revert and HISTORY with the relevant snapshot ID.

You can also use the --steps-back flag to revert a pipeline by specifying how many steps to move back. For example, to revert a pipeline named my-pipeline to its immediate previous configuration, run:

calyptia rollout pipeline my-pipeline --steps-back 1

Upgrade pipeline version

To upgrade a pipeline's underlying binary file, use the update pipeline command and the --image flag:

calyptia update pipeline PIPELINE --image ghcr.io/calyptia/core/calyptia-fluent-bit:VERSION

Replace PIPELINE with either the name or unique ID of the pipeline you'd like to modify and VERSION with the latest version of the Calyptia pipeline.

Modify pipeline network settings

After you deploy or update a pipeline that has a network-based source, Calyptia automatically load-balances that pipeline using Kubernetes services (even if you deployed Calyptia Core on a Linux server).

By default, Calyptia uses the LoadBalancer network service. You can modify this behavior by running the following command:

calyptia update pipeline PIPELINE --service-type SERVICE

Replace PIPELINE with either the name or unique ID of the pipeline whose network setting you'd like to modify and SERVICE with the service type you'd like to use. Possible service type values are LoadBalancer, NodePort, and ClusterIP.

Scale a pipeline

To scale a pipeline, use the --replicas flag:

calyptia update pipeline PIPELINE --replicas REPLICAS

Replace PIPELINE with either the name or unique ID of the pipeline you'd like to scale and REPLICAS with the number of replicas you'd like to create.

Horizontal pod autoscaling

You can also use horizontal pod autoscaling (HPA) to scale pipelines automatically.

For example, to enable HPA for an existing pipeline:

calyptia update pipeline PIPELINE --min-replicas 1 \
   --scale-up-type Pods --scale-up-period-seconds 30 --scale-up-value 2 \
   --scale-down-type Pods --scale-down-value 15 --scale-down-period-seconds 90 \
   --utilization-cpu-average 25 --utilization-memory-average 80

Replace PIPELINE with either the name or unique ID of the pipeline you'd like to scale.

Add pipeline files

By default, each pipeline includes a file that lists the parsers used by that pipeline. You can also add your own files to a pipeline, including certificates, stream processor jobs, and custom Lua scripts.

To add files to a pipeline, use the --file flag:

calyptia update pipeline PIPELINE --file FILENAME

Replace PIPELINE with either the name or unique ID of your pipeline and FILENAME with the name of the file to add. Files must be less than 10 MB and must not include a file extension.

After you've added a file, you can access it through the {{ files.FILENAME }} variable in configuration files and the Calyptia Core Dashboard.

Add encrypted file

To encrypt pipeline files, use the --encrypt-files flag. Calyptia uses RSA‑OAEP encryption with a 2048-bit key.

Add pipeline secrets

To add secrets to a pipeline, run the following command:

calyptia update pipeline --secrets-file FILE --secrets-format FORMAT

Replace FILE with the name of your secrets file and FORMAT with the format of that file. Possible format values are auto, env, json, and yaml.

Delete a pipeline

To delete a pipeline, run the following command:

calyptia delete pipeline PIPELINE

Replace PIPELINE with either the name or unique ID of the pipeline you'd like to delete.