Kafka source plugin
Apache Kafka is an open source distributed streaming platform that is used to collect, store, and process large volumes of real-time data streams. It's particularly well-suited for collecting telemetry data, which is the data produced by sensors and devices that monitor the performance and health of systems, machines, and applications.
Kafka is designed to handle high volumes of data in real time, making it a popular choice for applications that require low latency and high throughput. It uses a publish-subscribe model to collect and distribute data among multiple data producers and consumers. Data producers, such as sensors or applications, publish data to Kafka topics, while data consumers subscribe to these topics to receive the data.
You can use the Kafka source plugin to configure Chronosphere Telemetry Pipeline to collect data from your Apache Kafka instances.
Configuration parameters
The Kafka source plugin provides these configuration parameters:
Metadata
| Name | Key | Description | Default |
|---|---|---|---|
| Name | Name | Plugin instance name for identification purposes. | none |
General
| Key | Description |
|---|---|
| Brokers | Single of multiple lists of Kafka Brokers. For example, 192.168.1.3:9092, 192.168.1.4:9092. |
| Topics | Single entry or list of topics separated by comma (,) that Calyptia Core uses to receive messages from Kafka. |
Advanced
| Key | Description |
|---|---|
| Minimum Queued Messages | Minimum number of messages per topic+partition Calyptia Core tries to maintain in the local consumer queue. |
| Group ID | Client group ID string. All clients sharing the same group.id belong to the same group. |