Scaling

When ingesting data with network sources or agents, you may want to scale the pipeline to process additional records. With Calyptia Core, all ingested data is automatically load balanced within a pipeline replica.

Within Advanced Settings, increase the number of replicas to the desired amount.

To see these steps in real time, follow along with an interactive demo (opens in a new tab).

Calyptia CLI

Scaling up an existing Calyptia Core pipeline can be performed using the update command:

calyptia update pipeline PIPELINE_NAME_OR_ID --replicas NUM_OF_REPLICAS

To examine the number of replicas for a given pipeline, run:

calyptia get pipelines --core-instance CORE_INSTANCE

The output is similar to the following:

 NAME                 REPLICAS  STATUS   AGE
 pipeline-name               3  STARTED  15 hours

Horizontal pod autoscaling

Requires Core Instance version 2.12.3 or later.

Calyptia pipelines support horizontal pod autoscaling (opens in a new tab) (HPA), an optional feature that enables Calyptia to scale pipelines up or down as needed.

Enable horizontal pod autoscaling

To enable HPA, create or update a pipeline through Calyptia CLI and include both the --scale-up-value and --scale-down-value flags. You must assign a value greater than 0 to both flags, but these values don't need to be identical to each other.

Configure horizontal pod autoscaling

When you create or update a pipeline through Calyptia CLI, you can use the following flags to configure HPA:

• --min-replicas: the minimum number of replicas to which the autoscaler can scale down.
• --scale-up-type: the type of scale-up policy to implement. Possible values are Pods or Percent.
• --scale-up-value: (required) the amount of change to implement through scale-up policies. Must be greater than 0.
• --scale-up-period-seconds: the window of time for which the scale-up policy should hold true, in seconds.
• --scale-down-type: the type of scale-down policy to implement. Possible values are Pods or Percent.
• --scale-down-value: (required) the amount of change to implement through scale-down policies.
• --scale-down-period-seconds: the window of time for which the scale-down policy should hold true, in seconds.
• --utilization-cpu-average: the target percentage value for average CPU utilization. The autoscaler attempts to maintain equilibrium around this value by scaling pipelines up or down accordingly.
• --utilization-memory-average: the target percentage value for average memory utilization. The autoscaler attempts to maintain equilibrium around this value by scaling pipelines up or down accordingly.

Disable horizontal pod autoscaling

To disable HPA, update a pipeline through Calyptia CLI and set both the --scale-up-value and --scale-down-value flags to 0. This prevents the autoscaler from scaling pipelines up or down, which means other HPA settings won't affect your pipelines.