TELEMETRY PIPELINE

Release notes

Chronosphere Telemetry Pipeline release notes

Information for each release is displayed by component.

Version 2.20.0

Released: 2024.11.12

Web interface

Added more settings to the S3 Input (SQS) source plugin.

Pipeline agent

Fix: Reduced the memory usage of the block records processing rule.

Version 2.19.0

Released: 2024.11.05

Web interface

Various improvements to the fleet management interface.
Fix: Fixed an issue that made some processing rule templates unavailable in the Pipeline Builder.
Fix: Fixed an issue that hid pipelines' version numbers.

Core Operator

Added a host path volume to Daemonset pipelines.
Added tolerations to the Core Operator chart.

Version 2.18.0

Released: 2024.10.30

Web interface

Various improvements to the processing rules interface.
Added documentation links for fleets.
Added the channel parameter to the Splunk destination plugin.
Fix: Fixed several typos in the fleet onboarding walkthrough that could cause certain commands to fail.
Fix: Fixed an issue with the random sampling processing rule that could cause the web interface to crash and reload.

Pipeline agent

Fix: Fixed an issue that artificially limited the number of processing rules that could run in a single pipeline.
Fix: Fixed an issue with the aws_sqs_enable_imds parameter in the S3 Input (SQS) source plugin.

Core Operator

Fix: Fixed an issue that overwrote the standard label names provided by Kubernetes templates.

Version 2.17.0

Released: 2024.10.21

Web interface

Updated names and logos to match Chronosphere Telemetry Pipeline branding.
Added more descriptive error messages throughout the web interface.
Fix: Clarified the instructions for adding an agent to a fleet.
Fix: Fixed an issue that restricted the types of files that can be added to fleets.

Pipeline API

Pipelines with the Deployment PipelineKind now use the rolling update strategy instead of the recreate strategy.
Fix: Added backwards compatibility for fleet agents.

Pipeline agent

Added the Cloudflare source plugin.
Fix: Fixed an issue that caused 404 errors from the OpenTelemetry destination plugin.

Version 2.16.0

Released: 2024.10.08

Web interface

Introduced character limits to Core Instance and pipeline names to prevent name-related errors. Core Instance names are limited to 20 characters, and pipeline names are limited to 27 characters.
Added a character limit to regular expression GPT prompts. Only the first 16,834 characters of input text are included in prompts.
Added documentation links for each plugin.
Added the YAML configuration view to the Pipeline Builder.
Added the ability to rename processing rule templates.
Improved how confirmation dialogs appear on wide screens.
Improved how the processing rules dialog appears at lower resolutions.
Added detailed error messages when manually editing a YAML configuration in the Pipeline Builder. These messages now highlight line syntax errors.
Added a JSON linter to the processing rules dialog editor for JSON input.
Fix: Closing the processing rules dialog now resets it before the next use.
Fix: Metrics now display properly on each pipeline's details page.
Fix: Fixed an issue where changing a project name through Pipeline CLI could cause synchronization errors.

Pipeline CLI

Added the --kind flag to set the PipelineKind. The PipelineKind attribute determines whether a pipeline is run as a Deployment (opens in a new tab), which has a configurable number of replicas, or a DaemonSet (opens in a new tab), which runs an instance for every node in the corresponding Kubernetes cluster.
Added the --max-replicas flag, which lets you specify the upper limit of replicas for a pipeline.

Pipeline API

Added the filter option exclude_internal=true for retrieving Core Instances and pipelines. When retrieving Core Instances, using this option excludes health check pipelines from the pipelinesCount field. When retrieving pipelines, using this option excludes health check pipelines from results.
Fix: Standardized the data type for pipeline statuses.
Fix: Standardized internal values for querying metrics.
Fix: Pipeline checks are now deduplicated before entering storage. This resolves an issue that previously caused 500 errors.
Fix: Sending an invalid URL query string to the audit logs API now returns the proper HTTP status code.
Fix: Updated several API error messages to display the correct component names.
Fix: Pipeline name length errors now appear before a pipeline is deployed, not after.

Pipeline agent

Fix: The listen parameter in networking plugins now works correctly.
Fix: Updated dependencies for increased security.

Core Operator

Fix: Validate pipeline names to ensure they don't exceed 27 characters.

Version 2.15.0

Released: 2024.09.24

Web interface

Updated the names of source and destination plugins for consistency.
Updated the appearance of alert popups.
Changed the name of the Starting pipeline status to Running.
Added keyboard shortcuts to all confirmation modals. Users can now press the Enter key to confirm an action or the Escape key to cancel it.
Fix: Fixed a bug that prevented editing a pipeline whose configuration file included a match '*' filter.
Fix: Fixed pagination issues that affected the pipelines list and pipeline secrets list.
Fix: The processing rules modal now displays accurate stats.
Fix: Fixed a bug that affected the ability to save templates when creating a new pipeline.
Fix: Fixed an issue that made buttons inaccessible in the processing rules modal for certain screen resolutions.

Pipeline CLI

Added more descriptive error messages for errors related to Core Instance timeouts.
Fix: Fixed a bug that returned an ambiguous name error for resources with similar names.

Pipeline API

Fix: Fixed a bug where logs with invalid UTF-8 characters would cause a 500 server error. Null characters are now replaced with an empty string, and other invalid characters return the appropriate descriptive errors.
Fix: Metric labels are now deduplicated before entering storage.

Pipeline agent

Fix: Setting a region in the Amazon S3 SQS plugin now sets the corresponding endpoint automatically.
Fix: The aggregate records processing rule now handles unsigned integers correctly.

Version 2.14.0

Also known as Chronosphere Telemetry Pipeline v2.14.0.

Released: 2024.09.11

Web interface

Renamed the Teammates page to Team.
Core Instance tags can now include numbers.
Added a confirmation modal when pausing a pipeline.
Clarified the names of menu items in the Data Pipelines section of the Core Instance details page.
Added the Status History button to the More options menu in the Data Pipelines section of the Core Instance details page.
Updated the link to Chronosphere Support included in the pipeline Status History modal.
Added role tags to the project selector menu.
Added the pipeline builder playground.
Invalid cluster logging configurations now display an error message.
Added the Datagen source plugin to the pipeline builder.
Updated in-page links to the Chronosphere docs library.
Changed the behavior of the processing rules builder to load an empty state on startup.
Processing rules now support metrics and traces.
Fix: Fixed a bug that made processing rules disappear after updating a destination plugin's configuration settings.
Fix: Fixed a bug related to reserved words in Core Instance names.

Pipeline CLI

Added the ability to set proxy environment variables using the following flags: –cloud-proxy, –http_proxy, –https_proxy, and –no_proxy.
Added support for the --metrics flag to enable Kubernetes server metrics.
Removed the deprecated --auto-create-ports flag.
Added the --disable-auto-checks flag as an option when creating or updating pipelines.
Improved the performance of the autocomplete feature and the quality of its results.

Pipeline API

Added the new audit logs feature.
Users with read-only permissions (the Viewer role) can no longer create or access API tokens.
Processing rules now draw upon V2 of the template API.
The horizontal pod autoscaling API now supports the Unknown scaling policy.
Ports are now configured on a per-pipeline level.
Pipeline image validation now supports any image that follows the repo:tag syntax.
The validation process for pipeline configuration files now supports arrays in addition to comma-separated and space-separated strings.
Configuration files now return an error in response to unknown fields.
Updated the list of resources returned in search results.
Updated the performance of the list endpoint when including ?include=ports,files,secrets as a search query.
Added support for the exact parameter when performing searches.
Tags now support numbers and dashes.
Fix: The Metrics API now ignores empty metrics instead of returning an error.
Fix: Sending invalid AWS tokens to the AWS Marketplace API now returns the correct 422 HTTP status code.
Fix: The validation process for configuration files no longer returns an error for core-prefixed properties.

Pipeline agent

Added support for raw messages to the S3 Input (SQS) source plugin.
Processing rules now support metrics and traces.
Added the SQL DB source plugin.
Added proxy support to the calyptia plugin.
Fix: Fixed a fatal error that stopped agent execution during a hot reload.
Fix: Various improvements to the Splunk source plugin.

Core Operator

Added support for proxies.
Improved image version replacement logic for Helm charts.

Version 2.13.0

Also known as Chronosphere Telemetry Pipeline v2.13.0.

Released: 2024.06.12

Web interface

Added support for the Kubernetes Metadata input plugin and the Datadog input plugin.
The OpenTelemetry input plugin now sets Tag_From_URI to false by default.
Added a Help Center Menu to the sidebar with links to various Telemetry Pipeline and Fluent Bit resources.
Added an About section to settings with version information about different Telemetry Pipeline components.
Fix: Fixed an issue where switching to a new project would not hide pipelines from the previous project.

Pipeline CLI

Added parameters to configure horizontal pod autoscaling.
Fix: Fixed an issue that automatically reverted a pipeline's service type to LoadBalancer when updating a pipeline without providing a --service-type flag.

Pipeline API

Updated the plugins schema for pipeline configuration validation.
Fix: Added a detailed error response to handle duplicate pipeline aliases.
Fix: Metrics that are more than 30 minutes old are no longer ingested.
Fix: Improved the performance of the Pipeline API's pipeline endpoint.

Pipeline agent

Added a Datagen input plugin that uses Gofakeit (opens in a new tab) to generate simulated log data.
Fix: Fixed a potential security issue (Fluent Bit CVE-2024-4323) related to input requests.

Core Operator

Pipelines now support horizontal pod autoscaling.
Fix: Fixed an issue that prevented Core Instance from fetching more than 10 pipelines at a time.
Fix: Changed the way that Core Instance assigns names to pipelines. The previous naming schema was [pipeline_name]-[random_4_char_string], and the new naming schema is [core-instance-name]-[pipeline_name]

Version 2.12.3

Released: 2024.05.21

Pipeline agent

Fix: Fixed a potential security issue (Fluent Bit CVE-2024-4323) related to input requests.

Version 2.12.0

Released: 2024.05.15

Web interface

Added output plugins for Exabeam, Oracle Logging Analytics, Azure Log Ingestion API, GELF (Graylog), Clickhouse DB, Scalyr DataSet, and Chronosphere logs.
Various updates to pipeline-related UI elements, including plugin metrics, output logs, and load-balanced endpoints.
Improved the look and feel of the pipeline editor.
Processing rules now remain visible in the UI after you disable them.
Added the ability to disable automated logging.
Fix: Fixed an issue that prevented empty Core Instances from redirecting to the Create new instance page.
Fix: Fixed an issue with the configuration settings for the Okta System Logs plugin.
Fix: Fixed an issue with automated logging that caused a CrashLoopBackOff state.
Fix: Added Pipeline CLI installation instructions that were missing from the Core Instance install wizard.
Deprecated: Removed "Test connection" feature from the automate logging onboarding modal.
Deprecated: Removed Helm tags from Kubernetes aggregator onboarding instructions.
Deprecated: Removed Validate button from fleet management onboarding modal.

Pipeline CLI

Added the ability to set a tolerations parameter for new Core Instances, which sets pod tolerations to the Core Instance and propagates that value to pipeline replicas. For example:
```
calyptia create core_instance operator --name test8900 --tolerations key1=Exists:-:NoSchedule,key2=Equal:val2:PreferNoSchedule
```
Added the ability to set permissions for new members when sending invites to a project.

Pipeline API

Upgraded metrics backend to Prometheus.
Added the ability to immediately create a fleet with files by using the create endpoint.
Fix: Improved performance by adding SQL index to pipeline statuses and configuration.
Fix: Empty arrays now initialize to [] instead of null in the cluster object regular expressions API.
Fix: Fixed an error that prevented users from deleting fleets.
Deprecated: Removed the ability to query metrics through the v1 API.

Pipeline agent

Added the Scalyr DataSet output plugin.
Updated various dependencies.

Core Operator

Updated sync to accept the new tolerations parameter.
The Core Instance tolerations parameter is now propagated to pipeline replica sets.
Updated Helm chart to include the skipServiceCreation parameter. This parameter disables the creation of new Kubernetes services for all pipelines in the Core Instance. For example:
```
helm install ci calyptia/core-instance --set "skipServiceCreation=true" --set "coreInstance=12l34" --set "cloudToken=1l234"
```

Updated Helm chart to include the tolerations parameter. For example:

helm install ci core-instance/ --set "tolerations=key1=Equal:value1:Execute:3600\,key2=Exists:value2:NoExecute" --set "coreInstance=12l34" --set "cloudToken=1l234"

Fix: Fixed an issue with annotations flag parsing that affected comma-separated values.

Processing rules

Added multiple regular expression engine options for most processing rules that support regular expressions.
Updated the "parse" processing rule to support extended regular expression syntax.
Added the "join" processing rule.

Security Component versions