Scaling
When ingesting data with network sources or agents, you may want to scale the pipeline to process additional records. With Calyptia Core, all ingested data is automatically load balanced within a pipeline replica.
Within Advanced Settings, increase the number of replicas to the desired amount.
To see these steps in real time, follow along with an interactive demo (opens in a new tab).
For replica count recommendations over 100 TB a day, contact the Calyptia team prior to scaling up to ensure proper stability.
Calyptia CLI
Scaling up an existing Calyptia Core pipeline can be performed using the update command:
calyptia update pipeline PIPELINE_NAME_OR_ID --replicas NUM_OF_REPLICAS
To examine the number of replicas for a given pipeline, run:
calyptia get pipelines --core-instance CORE_INSTANCE
The output is similar to the following:
NAME REPLICAS STATUS AGE
pipeline-name 3 STARTED 15 hours
Horizontal pod autoscaling
Requires Core Instance version 2.12.3 or later.
Calyptia pipelines support horizontal pod autoscaling (opens in a new tab) (HPA), an optional feature that enables Calyptia to scale pipelines up or down as needed.
Enable horizontal pod autoscaling
To enable HPA, create or update a pipeline through Calyptia CLI
and include both the --scale-up-value
and --scale-down-value
flags. You must
assign a value greater than 0
to both flags, but these values don't need to be
identical to each other.
Configure horizontal pod autoscaling
When you create or update a pipeline through Calyptia CLI, you can use the following flags to configure HPA:
--min-replicas
: The minimum number of replicas to which the autoscaler can scale down.--scale-up-type
: The type of scale-up policy to implement. Possible values arePods
orPercent
.--scale-up-value
: Required. The amount of change to implement through scale-up policies. Must be greater than0
.--scale-up-period-seconds
: The window of time for which the scale-up policy should hold true, in seconds.--scale-down-type
:The type of scale-down policy to implement. Possible values arePods
orPercent
.--scale-down-value
: Required. The amount of change to implement through scale-down policies.--scale-down-period-seconds
: The window of time for which the scale-down policy should hold true, in seconds.--utilization-cpu-average
: The target percentage value for average CPU utilization. The autoscaler attempts to maintain equilibrium around this value by scaling pipelines up or down accordingly.--utilization-memory-average
: The target percentage value for average memory utilization. The autoscaler attempts to maintain equilibrium around this value by scaling pipelines up or down accordingly.
Disable horizontal pod autoscaling
To disable HPA, update a pipeline through Calyptia CLI
and set both the --scale-up-value
and --scale-down-value
flags to 0. This
prevents the autoscaler from scaling pipelines up or down, which means other HPA
settings won't affect your pipelines.