Resource profiles - Chronosphere Documentation

This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform.

Chronosphere Telemetry Pipeline uses templates called resource profiles to configure settings related to buffering and resource consumption. You can choose between several predefined resource profiles or create your own resource profiles that meet your requirements for performance and reliability.

To avoid the potential for data loss, avoid using improperly configured resource profiles or resource profiles incompatible with your environment, including profiles that exceed your available resources. Additionally, updating the resource profile for an existing pipeline causes that pipeline to update and potentially restart, which can interrupt the flow of data through that pipeline.

Predefined resource profiles

The following predefined resource profiles are available in Telemetry Pipeline:

best-effort-low-resource
high-performance-guaranteed-delivery
high-performance-optimal-throughput

If you don’t specify a resource profile when you create a pipeline, Telemetry Pipeline uses the best-effort-low-resource profile by default. Each resource profile uses the following values:

Type	Description
`best-effort-low-resource`	- `StorageMaxChunksUp`: `128` - `StorageSyncFull`: `false` - `StorageBacklogMemLimit`: `5M` - `StorageVolumeSize`: `250Mi` - `CPULimit`: `0` - `CPURequest`: `0` - `MemoryLimit`: `0` - `MemoryRequest`: `0`
`high-performance-guaranteed-delivery`	- `StorageMaxChunksUp`: `256` - `StorageSyncFull`: `false` - `StorageBacklogMemLimit`: `256M` - `StorageVolumeSize`: `512Mi` - `CPULimit`: `0` - `CPURequest`: `0` - `MemoryLimit`: `0` - `MemoryRequest`: `0`
`high-performance-optimal-throughput`	- `StorageMaxChunksUp`: `0` - `StorageSyncFull`: `false` - `StorageBacklogMemLimit`: `""` - `StorageVolumeSize`: `128Mi` - `CPULimit`: `0` - `CPURequest`: `0` - `MemoryLimit`: `0` - `MemoryRequest`: `0`

You can’t edit these predefined resource profiles, but you can define new resource profiles.

Settings

Each resource profile includes the following settings:

Name in Pipeline CLI	JSON configuration key	Description	Possible values
`STORAGE-SYNC-FULL`	`resources.storage.syncFull`	Determines how often the memory buffer and file system buffer are synchronized. If `true`, uses the `full` Fluent Bit `storage.sync` mode, which synchronizes after every write operation. If `false`, uses the `normal` mode, which synchronizes at regular intervals. This setting affects only StatefulSet pipelines. If unspecified, defaults to `false`.	`true`, `false`
`STORAGE-BACKLOG-MEM-LIMIT`	`resources.storage.backlogMemLimit`	Required. Sets the amount of memory allocated for storing buffered data. This setting only affects StatefulSet pipelines, but it’s still required in all resource profile definitions.	Memory in MB
`STORAGE-VOLUME-SIZE`	`resources.storage.volumeSize`	Required. Sets the size of the storage volume in each pipeline Pod. Pipelines with the Deployment or DaemonSet workload type use an ephemeral storage volume. Pipelines with the StatefulSet workload type use a persistent storage volume.	Volume size in Mi/Gi/Ti
`STORAGE-MAX-CHUNKS-UP`	`resources.storage.maxChunksUp`	Sets the maximum number of chunks that can be buffered in memory. This setting only affects StatefulSet pipelines. If unspecified, defaults to `0`.	Any integer greater than or equal to `0`
`CPU-LIMIT`	`resources.cpu.limit`	Required. Sets the CPU limit.	Any Kubernetes CPU value
`CPU-REQUEST`	`resources.cpu.request`	Required. Sets the CPU request.	Any Kubernetes CPU value
`MEM-LIMIT`	`resources.memory.limit`	Required. Sets the memory limit.	Any Kubernetes memory value
`MEM-REQUEST`	`resources.memory.request`	Required. Sets the memory request.	Kubernetes memory value

A value of 0 sets an unlimited upper bound.

Pipeline CLI commands

Use the following Pipeline CLI commands to interact with resource profiles.

Specify the resource profile for a new pipeline

Set the resource profile for a new pipeline by using the following command:

calyptia create pipeline --core-instance INSTANCE --name PIPELINE --resource-profile PROFILE

Replace the following:

INSTANCE: The name of your Core Instance.
PIPELINE: A name for your new pipeline.
PROFILE: The name of the resource profile to use.

View the resource profile of an existing pipeline

Use the following command to view the active resource profile of an existing pipeline:

calyptia get pipeline PIPELINE --output-format yaml

Replace PIPELINE with the name of your pipeline. In the resulting YAML output, the resourceProfile key contains details about the pipeline’s active resource profile, including its name and settings.

Update the resource profile for an existing pipeline

Update the resource profile for an existing pipeline by using the following command:

calyptia update pipeline PIPELINE --resource-profile PROFILE

Replace the following:

PIPELINE: The name or unique ID of your pipeline.
PROFILE: The name of the resource profile to use.

Define a new resource profile

Use the following command to define your own resource profiles in Telemetry Pipeline:

calyptia create resource_profile --core-instance INSTANCE --name PROFILE --spec FILE_NAME

Replace the following:

INSTANCE: The name of your Core Instance.
PROFILE: A name for your new resource profile.
FILE_NAME: The JSON file that defines your resource profile.

This is an example of a resource profile with a valid JSON structure:

{
  "resources": {
    "storage": {
      "syncFull": false,
      "backlogMemLimit": "5M",
      "volumeSize": "250Mi",
      "maxChunksUp": 128
    },
    "cpu": {
      "limit": "0",
      "request": "0"
    },
    "memory": {
      "limit": "0",
      "request": "0"
    }
  }
}

Change the settings of a resource profile

Requires Chronosphere Telemetry Pipeline v2.48.0 or later.

Use the following command to change the settings of an existing resource profile:

calyptia update resource_profile --core-instance INSTANCE --name PROFILE --spec FILE_NAME

Replace the following:

INSTANCE: The name of your Core Instance.
PROFILE: The name of the resource profile to update.
FILE_NAME: The JSON file that defines the new settings for your resource profile.

If any pipelines are using the specified resource profile, changes to the resource profile are applied to those pipelines accordingly.

List available resource profiles

Use the following command to list the resource profiles associated with a specified Core Instance:

calyptia get resource_profiles --core-instance INSTANCE

Replace INSTANCE with the name of the Core Instance. You can also add the --show-ids flag to print the unique IDs associated with each resource profile.

Get details about a resource profile

Use the following command to get details about a specific resource profile:

calyptia get resource_profile ID

Replace ID with the unique ID of that resource profile.

Delete a resource profile

Use the following command to delete a specific resource profile:

calyptia delete resource_profile ID

Replace ID with the unique ID of that resource profile.

​Predefined resource profiles

​Settings

​Pipeline CLI commands

​Specify the resource profile for a new pipeline

​View the resource profile of an existing pipeline

​Update the resource profile for an existing pipeline

​Define a new resource profile

​Change the settings of a resource profile

​List available resource profiles

​Get details about a resource profile

​Delete a resource profile

Predefined resource profiles

Settings

Pipeline CLI commands

Specify the resource profile for a new pipeline

View the resource profile of an existing pipeline

Update the resource profile for an existing pipeline

Define a new resource profile

Change the settings of a resource profile

List available resource profiles

Get details about a resource profile

Delete a resource profile