Scaling

When ingesting data with network sources or agents, you may want to scale the pipeline to process additional records. With Calyptia Core, all ingested data is automatically load balanced within a pipeline replica.

Within Advanced Settings, increase the number of replicas to the desired amount.

To see these steps in real time, follow along with an interactive demo (opens in a new tab).

For replica count recommendations over 100 TB a day, contact the Calyptia team prior to scaling up to ensure proper stability.

Calyptia CLI

Scaling up an existing Calyptia Core pipeline can be performed using the update command:

calyptia update pipeline PIPELINE_NAME_OR_ID --replicas NUM_OF_REPLICAS

To examine the number of replicas for a given pipeline, run:

calyptia get pipelines --core-instance CORE_INSTANCE

The output is similar to the following:

 NAME                 REPLICAS  STATUS   AGE
 pipeline-name               3  STARTED  15 hours