Amazon Kinesis Streams destination plugin
The Amazon Kinesis Streams destination plugin lets you stream and analyze your data by sending it directly to your Amazon Kinesis stream.
Configuration parameters
The Amazon Kinesis Streams destination plugin provides these configuration parameters. Items in the Name column display in the Telemetry Pipeline web interface. Items in the Key column are the YAML keys to use in pipeline configuration files.
Required
| Name | Key | Description | Default |
|---|---|---|---|
| Region | region | Required. The AWS region of your Kinesis Streams. | us-east-1 |
| Kinesis Stream Name | stream | Kinesis Stream Name | none |
AWS Authentication
| Name | Key | Description | Default |
|---|---|---|---|
| IAM Role ARN | role_arn | ARN of an IAM role to assume. For example, for cross account access. | none |
| Kinesis Streams API Endpoint | endpoint | Custom endpoint for the Kinesis API. | none |
| STS API Endpoint | sts_endpoint | Custom endpoint for the STS API. | none |
| External ID for STS API | external_id | Specify an external ID for the STS API. Can be used with the role_arn parameter if your role requires an external ID. | none |
Advanced
| Name | Key | Description | Default |
|---|---|---|---|
| Time Key | time_key | Add the timestamp to the record under this key. By default, the timestamp from Fluent Bit isn't added to records sent to Kinesis. | none |
| Time Key Format | time_key_format | strftime-compliant format string for the timestamp. This option is used with time_key. | %Y-%m-%dT%H:%M:%S |
| Log Key | log_key | By default, the entire log record is sent to Kinesis. If you specify a key name with this option, only the value of that key is sent to Kinesis. For example, if you're using the Fluentd Docker log driver, you can specify log_key log, and only the log message is sent to Kinesis. | none |
| Enable Auto Retry Requests | auto_retry_requests | Immediately retry failed requests to AWS services once. This option doesn't affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient or random networking issues. | false |
Security and TLS
| Name | Key | Description | Default |
|---|---|---|---|
| TLS | tls | Enable or disable TLS/SSL support. Accepted values: true, false. | false |
| TLS Certificate Validation | tls.verify | Turn TLS/SSL certificate validation on or off. TLS must be on for this setting to be enabled. | on |
| TLS Debug Level | tls.debug | Set TLS debug verbosity level. Accepted values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), 4 (Verbose). | 1 |
| CA Certificate File Path | tls.ca_file | Absolute path to CA certificate file. | none |
| Certificate File Path | tls.crt_file | Absolute path to certificate file. | none |
| Private Key File Path | tls.key_file | Absolute path to private key file. | none |
| Private Key Path Password | tls.key_passwd | Optional password for tls.key_file file. | none |
| TLS SNI Hostname Extension | tls.vhost | Hostname to be used for TLS SNI extension. | none |
Advanced Networking
| Name | Key | Description | Default |
|---|---|---|---|
| DNS Mode | net.dns.mode | Select the primary DNS connection type, which can be TCP or UDP. | none |
| DNS Resolver | net.dns.resolver | Select the primary DNS connection type, which can be LEGACY or ASYNC. | none |
| Prefer IPv4 | net.dns.prefer_ipv4 | Prioritize IPv4 DNS results when trying to establish a connection. Accepted values: true, false. | false |
| Keepalive | net.keepalive | Enable or disable Keepalive support. Accepted values: true, false. | true |
| Keepalive Idle Timeout | net.keepalive_idle_timeout | Set maximum time allowed for an idle Keepalive connection. | 30s |
| Max Connect Timeout | net.connect_timeout | Set maximum time allowed to establish a connection, which includes the TLS handshake. | 10s |
| Max Connect Timeout Log Error | net.connect_timeout_log_error | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. Accepted values: true, false. | true |
| Max Keepalive Recycle | net.keepalive_max_recycle | Set maximum number of times a keepalive connection can be used before it's retired. | 2000 |
| Source Address | net.source_address | Specify network address to bind for data traffic. | none |