Kafka destination plugin
The Kafka destination plugin lets you configure Chronosphere Telemetry Pipeline to integrate your data with Apache Kafka for real-time processing and analysis.
Supported telemetry types
This plugin supports these telemetry types:
| Logs | Metrics | Traces |
|---|---|---|
Configuration parameters
Use the parameters in this section to configure your plugin. The Telemetry Pipeline web interface uses the values in the Name column to describe the parameters. Items in the Key column are the YAML keys to use in pipeline configuration files.
General
| Name | Key | Description | Default |
|---|---|---|---|
| Brokers | brokers | Required. Single of multiple list of Kafka brokers. For example, 192.168.1.3:9092 or 192.168.1.4:9092. | none |
| Topics | topics | Required. Single entry or list of comma-separated topics that Telemetry Pipeline uses to send messages to Kafka. If only one topic is set, that topic is used for all records. If multiple topics exists, the one set in the record by topic_key is used. | calyptia-core |
| Format | format | Specify data format. Accepted values: json, msgpack. | json |
Advanced
| Name | Key | Description | Default |
|---|---|---|---|
| Message Key | message_key | Optional key to store the message. | none |
| Message Key Field | message_key_field | If set, the value of message_key_field in the record will indicate the message key. If not set or found in the record, message_key will be used (if set). | none |
| Timestamp Key | timestamp_key | Set the key to store the record timestamp. | none |
| Timestamp Format | timestamp_format | Set the format of the timestamp. Accepted values: double, iso8601. | double |
| Body Key | body_key | Specify the key which contains the body. | none |
| Queue Full Retries | queue_full_retries | Telemetry Pipeline queues data into rdkafka library. If the underlying library can't flush the records, the queue might fill up, blocking new addition of records. This key sets the number of local retries to enqueue the data. The interval between each retry is one second. Setting the key to 0 sets an unlimited number of retries. | 10 |