Scaling and replicas
Chronosphere Telemetry Pipeline uses replica pods to scale pipelines. Each pipeline needs at least one replica to function. If a pipeline has more than one replica, any ingested data is balanced between those replicas accordingly.
You can assign a fixed number of replicas to a pipeline, or enable horizontal pod autoscaling to automatically add and remove replicas in response to utilization changes.
Fixed number of replicas
Use one of the following methods to set a fixed number of replicas for your pipelines:
For replica counts with a volume over 100 TB a day, contact Chronosphere Support prior to scaling up to ensure stability.
- Sign in to the Telemetry Pipeline web interface (opens in a new tab).
- Open the project that contains the pipeline that you want to modify.
- Go to Core Instances, then click the name of the Core Instance where you deployed the pipeline that you want to modify.
- Under Data Pipelines, click the name of the pipeline that you want to modify.
- Go to Advanced Settings.
- Increase the Replicas field to the desired amount, and then click Apply.
Horizontal pod autoscaling
Requires Core Instance version 2.12.3 or later.
Pipelines support horizontal pod autoscaling (opens in a new tab) (HPA), which enables Telemetry Pipeline to add and remove replicas as needed.
Enable horizontal pod autoscaling
To enable HPA, create or modify a pipeline
through Pipeline CLI and include both the --scale-up-value and
--scale-down-value flags. You must assign a value greater than 0 to both flags,
but these values don't need to be identical to each other.
Configure horizontal pod autoscaling
To configure HPA, use the following flags when you create or update a pipeline through Pipeline CLI:
| Flag | Description | Value |
|---|---|---|
--max-replicas | The maximum number of replicas to which the autoscaler can scale up. | Example: 6 |
--min-replicas | The minimum number of replicas to which the autoscaler can scale down. | Example: 3 |
--scale-up-type | The type of scale-up policy to implement. | Pods orPercentz |
--scale-up-value | Required. The amount of change to implement through scale-up policies. | Must be greater than 0. |
--scale-up-period-seconds | The window of time for which the scale-up policy should hold true, in seconds. | Example: 30 |
--scale-down-type | The type of scale-down policy to implement. | Pods or Percent |
--scale-down-value | Required. The amount of change to implement through scale-down policies. | Example: 15 |
--scale-down-period-seconds | The window of time for which the scale-down policy should hold true, in seconds. | Example: 30 |
--utilization-cpu-average | The target percentage value for average CPU utilization. The autoscaler attempts to maintain equilibrium around this value by scaling pipelines up or down accordingly. | Example: 25 |
--utilization-memory-average | The target percentage value for average memory utilization. The autoscaler attempts to maintain equilibrium around this value by scaling pipelines up or down accordingly. | Example: 30 |
Disable horizontal pod autoscaling
To disable HPA, update a pipeline through Pipeline CLI
and set both the --scale-up-value and --scale-down-value flags to 0. This
prevents the autoscaler from scaling pipelines up or down, which means other HPA
settings won't affect your pipelines.
Pause a pipeline
You can pause a pipeline by setting its replicas to zero, which temporarily prevents that pipeline from routing or processing data. Use one of these methods to pause a pipeline:
- Sign in to Telemetry Pipeline (opens in a new tab).
- Open the project that contains the pipeline that you want to pause.
- Go to Core Instances, then click the name of the Core Instance where you deployed the pipeline that you want to pause.
- Under Data Pipelines, find the pipeline that you want to pause, and then click Settings > Pause.
- In the dialog that appears, click Yes to confirm your choice.
Unpause a pipeline
If a pipeline is paused, use one of these methods to unpause it:
- Sign in to Telemetry Pipeline (opens in a new tab).
- Open the project that contains the pipeline that you want to unpause.
- Go to Core Instances, then click the name of the Core Instance where you deployed the pipeline that you want to unpause.
- Under Data Pipelines, find the pipeline that you want to unpause, and then click Settings > Resume.
- In the dialog that appears, click Yes to confirm your choice.