Resource profiles

TELEMETRY PIPELINE

Resource profiles

Chronosphere Telemetry Pipeline uses templates called resource profiles to configure settings related to buffering and resource consumption. You can choose between several predefined resource profiles or create your own resource profiles that meet your requirements for performance and reliability.

⚠️

To avoid the potential for data loss, avoid using improperly configured resource profiles or resource profiles incompatible with your environment, including profiles that exceed your available resources. Additionally, updating the resource profile for an existing pipeline causes that pipeline to update and potentially restart, which can interrupt the flow of data through that pipeline.

Predefined resource profiles

The following predefined resource profiles are available in Telemetry Pipeline:

best-effort-low-resource
high-performance-guaranteed-delivery
high-performance-optimal-throughput

If you don’t specify a resource profile when you create a pipeline, Telemetry Pipeline uses the best-effort-low-resource profile by default.

Each resource profile uses the following values:

Type	Description
`best-effort-low-resource`	- `StorageMaxChunksUp`: `128` - `StorageSyncFull`: `false` - `StorageBacklogMemLimit`: `5M` - `StorageVolumeSize`: `250Mi` - `CPULimit`: `0` - `CPURequest`: `0` - `MemoryLimit`: `0` - `MemoryRequest`: `0`
`high-performance-guaranteed-delivery`	- `StorageMaxChunksUp`: `256` - `StorageSyncFull`: `false` - `StorageBacklogMemLimit`: `256M` - `StorageVolumeSize`: `512Mi` - `CPULimit`: `0` - `CPURequest`: `0` - `MemoryLimit`: `0` - `MemoryRequest`: `0`
`high-performance-optimal-throughput`	- `StorageMaxChunksUp`: `0` - `StorageSyncFull`: `false` - `StorageBacklogMemLimit`: `""` - `StorageVolumeSize`: `128Mi` - `CPULimit`: `0` - `CPURequest`: `0` - `MemoryLimit`: `0` - `MemoryRequest`: `0`

You can’t edit these predefined resource profiles, but you can define new resource profiles.

Settings

Each resource profile includes the following settings:

Name in Pipeline CLI	JSON configuration key	Description	Possible values
`STORAGE-SYNC-FULL`	`resources.storage.syncFull`	If `true`, uses the `full` Fluent Bit `storage.sync` mode (opens in a new tab). If `false`, uses the `normal` mode. This setting affects only StatefulSet pipelines. If unspecified, defaults to `false`.	`true`, `false`
`STORAGE-BACKLOG-MEM-LIMIT`	`resources.storage.backlogMemLimit`	Required. Sets the maximum memory size used for the backlog queue when storing data chunks that cannot be sent to output. This setting only affects StatefulSet pipelines, but it’s still required in all resource profile definitions.	Memory in MB
`STORAGE-VOLUME-SIZE`	`resources.storage.volumeSize`	Required. Sets the size of the volume to mount in Pods for local storage processing.	Volume size in Mi/Gi/Ti
`STORAGE-MAX-CHUNKS-UP`	`resources.storage.maxChunksUp`	Sets the maximum number of data chunks that can be buffered in memory before being flushed to the filesystem. This setting only affects StatefulSet pipelines. If unspecified, defaults to `0`.	Any integer greater than or equal to `0`
`CPU-LIMIT`	`resources.cpu.limit`	Required. Sets the CPU limit (opens in a new tab).	Any Kubernetes CPU value (opens in a new tab)
`CPU-REQUEST`	`resources.cpu.request`	Required. Sets the CPU request (opens in a new tab).	Any Kubernetes CPU value (opens in a new tab)
`MEM-LIMIT`	`resources.memory.limit`	Required. Sets the memory limit (opens in a new tab).	Any Kubernetes memory value (opens in a new tab)
`MEM-REQUEST`	`resources.memory.request`	Required. Sets the memory request (opens in a new tab).	Kubernetes memory value (opens in a new tab)

A value of 0 sets an unlimited upper bound.

Pipeline CLI commands

Use the following Pipeline CLI commands to interact with resource profiles.

Specify the resource profile for a new pipeline

Set the resource profile for a new pipeline by using the following command:

calyptia create pipeline --core-instance INSTANCE --name PIPELINE --resource-profile PROFILE

Replace the following:

INSTANCE: The name of your Core Instance.
PIPELINE: A name for your new pipeline.
PROFILE: The name of the resource profile to use.

Update the resource profile for an existing pipeline

Update the resource profile for an existing pipeline by using the following command:

calyptia update pipeline --name PIPELINE --resource-profile PROFILE

Replace the following:

PIPELINE: The name or unique ID of your pipeline.
PROFILE: The name of the resource profile to use.

Define a new resource profile

Use the following command to define your own resource profiles in Telemetry Pipeline:

calyptia create resource_profile --core-instance INSTANCE --name PROFILE --spec FILE_NAME

Replace the following:

INSTANCE: The name of your Core Instance.
PROFILE: A name for your new resource profile.
FILE_NAME: The JSON file that defines your resource profile.

This is an example of a resource profile with a valid JSON structure:

{
 "resources": {
  "storage": {
   "syncFull": false,
   "backlogMemLimit": "5M",
   "volumeSize": "250Mi",
   "maxChunksUp": 128,
  },
  "cpu": {
   "limit": "0",
   "request": "0",
  },
  "memory": {
   "limit": "0",
   "request": "0"
  }
 }
}

List available resource profiles

Use the following command to list the resource profiles associated with a specified Core Instance:

calyptia get resource_profiles --core-instance INSTANCE

Replace INSTANCE with the name of the Core Instance.

You can also add the --show-ids flag to print the unique IDs associated with each resource profile.

Get details about a resource profile

Use the following command to get details about a specific resource profile:

calyptia get resource_profile ID

Replace ID with the unique ID of that resource profile.

Delete a resource profile

Use the following command to delete a specific resource profile:

calyptia delete resource_profile ID

Replace ID with the unique ID of that resource profile.

Pipeline configuration files Scaling and replicas