Kafka destination plugin

Apache Kafka is a distributed streaming platform used for building real-time data pipelines and streaming applications. It provides a high-throughput, low-latency platform for processing and transmitting large amounts of data, including logs, metrics, events, and other types of telemetry data.

This plugin provides a flexible and configurable way to transmit your data to Kafka. Whether you're working with logs, metrics, traces, or other types of telemetry data, the Kafka destination plugin in Calyptia Core provides a reliable and efficient way to integrate your data with Kafka for real-time processing and analysis.

Configuration parameters

The Kafka destination plugin provides these configuration parameters.


BrokersSingle of multiple list of Kafka Brokers. For example, or
TopicsSingle entry or list of topics separated by comma (,) that Fluent Bit will use to send messages to Kafka. If only one topic is set, that one will be used for all records. Instead if multiple topics exists, the one set in the record by Topic_Key will be used.
FormatSpecify data format, options available: json, msgpack.


FormatSpecify data format, options available: json, msgpack.
Message KeyOptional Key to store the message.
Timestamp KeySet the key to store the record timestamp.
Timestamp FormatSet the format to iso8601 or double.
Body KeySpecify the key which contains the body.
Queue Full RetriesFluent Bit queues data into rdkafka library, if for some reason the underlying library cannot flush the records the queue might fills up blocking new addition of records. The queue_full_retries option set the number of local retries to enqueue the data. The default value is 10 times, the interval between each retry is 1 second. Setting the queue_full_retries value to 0 sets an unlimited number of retries.
Message Key FieldIf set, the value of Message_Key_Field in the record will indicate the message key. If not set nor found in the record, Message_Key will be used (if set).