Amazon Kinesis Firehose destination plugin
The Amazon Kinesis Firehose destination plugin lets you configure Chronosphere Telemetry Pipeline to collect various data types such as logs, metrics, traces, and events and stream them directly to your Kinesis Firehose delivery stream.
Supported telemetry types
This plugin supports these telemetry types:
| Logs | Metrics | Traces |
|---|---|---|
Configuration parameters
Use the parameters in this section to configure your plugin. The Telemetry Pipeline web interface uses the values in the Name column to describe the parameters. Items in the Key column are the YAML keys to use in pipeline configuration files.
Required
| Name | Key | Description | Default |
|---|---|---|---|
| Region | region | Required. The AWS region to which to send your data. | us-east-1 |
| Firehouse delivery stream name | delivery_stream | Kinesis Firehouse name. | none |
AWS Authentication
| Name | Key | Description | Default |
|---|---|---|---|
| IAM Role ARN | role_arn | ARN of an IAM role to assume. For example, for cross account access. | none |
| Firehose API Endpoint | endpoint | Custom endpoint for the Firehose API. | none |
| STS API Endpoint | sts_endpoint | Custom endpoint for the STS API. | none |
| External ID for STS API | external_id | Specifies an external ID for the STS API. Can be used with the role_arn parameter if your role requires an external ID. | none |
Advanced
| Name | Key | Description | Default |
|---|---|---|---|
| Compression | compression | Compression type for Firehose records. Each log record is individually compressed and sent to Firehose. Accepted values: gzip, arrow (with arrow available only if Apache Arrow is enabled at compile time). | none |
| Time Key | time_key | Adds the timestamp to the record under this key. By default, the timestamp from Telemetry Pipeline isn't added to records sent to Kinesis. | none |
| Time Key Format | time_key_format | strftime-compliant format string for the timestamp. This option is used with time_key. | %Y-%m-%dT%H:%M:%S |
| Log Key | log_key | By default, the entire log record is sent to Kinesis. If you specify a key name with this option, only the value of that key is sent to Kinesis. For example, if you're using the Fluentd Docker log driver, you can specify log_key log, and only the log message is sent to Kinesis. | none |
| Enable Auto Retry Requests | auto_retry_requests | Whether to immediately retry failed requests to AWS services one time. This option doesn't affect the normal Telemetry Pipeline retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient or random networking issues. Accepted values: true, false. | false |
Security and TLS
| Name | Key | Description | Default |
|---|---|---|---|
| TLS | tls | Enables or disables TLS/SSL support. Accepted values: true, false. | false |
| TLS Certificate Validation | tls.verify | Enables or disables TLS/SSL certificate validation. TLS must be enabled for certificates to be validated. Accepted values: off, on. | on |
| TLS Debug Level | tls.debug | Sets TLS debug verbosity level. Accepted values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), 4 (Verbose). | 1 |
| CA Certificate File Path | tls.ca_file | Absolute path to CA certificate file. | none |
| Certificate File Path | tls.crt_file | Absolute path to certificate file. | none |
| Private Key File Path | tls.key_file | Absolute path to private key file. | none |
| Private Key Path Password | tls.key_passwd | Password for private key file. | none |
| TLS SNI Hostname Extension | tls.vhost | Hostname to be used for TLS SNI extension. | none |
Advanced Networking
| Name | Key | Description | Default |
|---|---|---|---|
| DNS Mode | net.dns.mode | Selects the primary DNS connection type, which can be TCP or UDP. | none |
| DNS Resolver | net.dns.resolver | Selects the primary DNS connection type, which can be LEGACY or ASYNC. | none |
| Prefer IPv4 | net.dns.prefer_ipv4 | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: true, false. | false |
| Keepalive | net.keepalive | Enables or disables Keepalive support. Accepted values: true, false. | true |
| Keepalive Idle Timeout | net.keepalive_idle_timeout | Sets the maximum time allowed for an idle Keepalive connection. | 30s |
| Max Connect Timeout | net.connect_timeout | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | 10s |
| Max Connect Timeout Log Error | net.connect_timeout_log_error | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: true, false. | true |
| Max Keepalive Recycle | net.keepalive_max_recycle | Sets the maximum number of times a keepalive connection can be used before it's retired. | 2000 |
| Source Address | net.source_address | Specifies the network address to bind for data traffic. | none |