> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Kafka source plugin

export const entity_0 = "Kafka source plugin"

export const plugin_0 = "Kafka source plugin"

You can use the Kafka [source plugin](/ingest/pipeline/plugins/source-plugins)
(name: `kafka`, alias: `kafka_input`) to ingest data from your Apache Kafka
instances into a telemetry pipeline.

This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin.

## Supported telemetry types

The {plugin_0} for Chronosphere Telemetry Pipeline supports these telemetry types:

|                    Logs                    |             Metrics             |              Traces             |
| :----------------------------------------: | :-----------------------------: | :-----------------------------: |
| <Icon icon="circle-check" color="green" /> | <Icon icon="ban" color="red" /> | <Icon icon="ban" color="red" /> |

## Configuration parameters

Use the parameters in this section to configure the {entity_0}. The
Telemetry Pipeline web interface uses the items in the **Name** column to
describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files)
use the items in the **Key** column as YAML keys.

### General

| Name        | Key       | Description                                                                                                                    | Default |
| ----------- | --------- | ------------------------------------------------------------------------------------------------------------------------------ | ------- |
| **Brokers** | `brokers` | Required. Single of multiple lists of Kafka brokers. For example, `192.168.1.3:9092`, `192.168.1.4:9092`.                      | *none*  |
| **Topics**  | `topics`  | Required. Single entry or list of topics separated by comma (`,`) that Telemetry Pipeline uses to receive messages from Kafka. | *none*  |

### Advanced

| Name                        | Key                           | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Default |
| --------------------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| **Minimum Queued Messages** | `rdkafka.queued.min.messages` | Minimum number of messages per topic and partition Telemetry Pipeline tries to maintain in the local consumer queue.                                                                                                                                                                                                                                                                                                                                            | `10`    |
| **Group ID**                | `rdkafka.group.id`            | The client group ID string. All clients with the same group ID belong to the same group.                                                                                                                                                                                                                                                                                                                                                                        | *none*  |
| **Memory Buffer Limit**     | `mem_buf_limit`               | For pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type only. Sets a limit for how much buffered data the plugin can write to memory, which affects [backpressure](/ingest/pipeline/v2/configure/backpressure). This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced. | *none*  |

### Other

This parameter doesn't have an equivalent setting in the Telemetry Pipeline web
interface, but you can use it in pipeline configuration files.

| Name   | Key               | Description                                                                                                                                                                                  | Default |
| ------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| *none* | `buffer_max_size` | Sets the maximum [chunk](https://docs.fluentbit.io/manual/administration/buffering-and-storage#chunks) size for buffered data. If a single log exceeds this size, the plugin drops that log. | `4M`    |

### Extended librdkafka parameters

This plugin uses the [librdkafka](https://github.com/confluentinc/librdkafka)
library. Certain configuration parameters available through the Telemetry
Pipeline UI are based on librdkafka settings. These parameters generally use the
`rdkafka.` prefix.

In addition to the parameters available through the Telemetry Pipeline UI, you can
customize any of the
[librdkafka configuration properties](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md)
by adding them to a pipeline configuration file. To do so, append the `rdkafka.`
prefix to the name of that property.

For example, to customize the `socket.keepalive.enable` property, add the
`rdkafka.socket.keepalive.enable` key to your configuration file.

<Note>
  Don't use librdkafka properties to configure a pipeline's memory buffer. Instead,
  use the [`buffer_max_size`](#other) parameter.
</Note>