Kafka destination plugin
Apache Kafka is a distributed streaming platform used for building real-time data pipelines and streaming applications. It provides a high-throughput, low-latency platform for processing and transmitting large amounts of data, including logs, metrics, events, and other types of telemetry data.
This plugin provides a flexible and configurable way to transmit your data to Kafka. Whether you're working with logs, metrics, traces, or other types of telemetry data, the Kafka destination plugin in Calyptia Core provides a reliable and efficient way to integrate your data with Kafka for real-time processing and analysis.
Configuration parameters
The Kafka destination plugin provides these configuration parameters.
General
| Key | Description |
|---|---|
| Brokers | Single of multiple list of Kafka Brokers. For example, 192.168.1.3:9092 or 192.168.1.4:9092. |
| Topics | Single entry or list of topics separated by comma (,) that Fluent Bit will use to send messages to Kafka. If only one topic is set, that one will be used for all records. Instead if multiple topics exists, the one set in the record by Topic_Key will be used. |
| Format | Specify data format, options available: json, msgpack. |
Advanced
| Key | Description |
|---|---|
| Format | Specify data format, options available: json, msgpack. |
| Message Key | Optional Key to store the message. |
| Timestamp Key | Set the key to store the record timestamp. |
| Timestamp Format | Set the format to iso8601 or double. |
| Body Key | Specify the key which contains the body. |
| Queue Full Retries | Fluent Bit queues data into rdkafka library, if for some reason the underlying library cannot flush the records the queue might fills up blocking new addition of records. The queue_full_retries option set the number of local retries to enqueue the data. The default value is 10 times, the interval between each retry is 1 second. Setting the queue_full_retries value to 0 sets an unlimited number of retries. |
| Message Key Field | If set, the value of Message_Key_Field in the record will indicate the message key. If not set nor found in the record, Message_Key will be used (if set). |