Amazon Kinesis Streams

Amazon Kinesis Streams destination plugin

Amazon Kinesis Streams is a highly scalable and durable real-time data streaming service provided by Amazon Web Services (AWS). The Kinesis Stream Destination Plugin in Calyptia Core allows you to stream and analyze your data by sending it directly to your Amazon Kinesis stream. With this plugin, you can configure your pipeline to collect various data types such as logs, metrics, traces, and events, and stream them to your Kinesis stream for real-time analysis and processing. The Kinesis Stream Destination Plugin provides a flexible and customizable way to integrate your data with your Kinesis stream, letting you tailor your streaming and analysis strategies to meet your specific needs.

Configuration parameters

The Amazon Kinesis Streams destination plugin provides these configuration parameters.


RegionThe AWS region of your Kinesis Streams.
Kinesis Stream NameKinesis Stream Name

AWS authentication

IAM Role ARNARN of an IAM role to assume (ex. for cross account access).
Kinesis API EndpointCustom Endpoint for the Kinesis API.
STS API EndpointCustom Endpoint for the STS API.
External ID for STS APISpecify an external ID for the STS API, can be used with the role_arn parameter if your role requires an external ID.


Time KeyAdd the timestamp to the record under this key. By default the timestamp from Fluent Bit will not be added to records sent to Kinesis.
Time Key Formatstrftime compliant format string for the timestamp; for example, the default is %Y-%m-%dT%H:%M:%S. This option is used with time_key.
Log KeyBy default, the whole log record will be sent to Kinesis. If you specify a key name with this option, then only the value of that key will be sent to Kinesis. For example, if you are using the Fluentd Docker log driver, you can specify log_key log and only the log message will be sent to Kinesis.
Enable Auto Retry RequestsImmediately retry failed requests to AWS services once. This option does not affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which may help improve throughput when there are transient/random networking issues.

Security and TLS

TLSEnable or disable TLS/SSL support.
TLS Certificate ValidationTurn TLS/SSL certificate validation on or off. TLS must be on for this setting to be enabled.
TLS Debug LevelSet TLS debug verbosity level. Accepts these values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), 4 (Verbose).
CA Certificate File PathAbsolute path to CA certificate file.
Certificate File PathAbsolute path to certificate file.
Private key File PathAbsolute path to private key file.
Private Key Path PasswordOptional password for tls.key_file file.
TLS SNI Hostname ExtensionHostname to be used for TLS SNI extension.

Advanced networking

DNS ModeSelect the primary DNS connection type (TCP or UDP).
DNS ResolverSelect the primary DNS connection type (TCP or UDP).
Prefer IPv4Prioritize IPv4 DNS results when trying to establish a connection.
KeepaliveEnable or disable Keepalive support.
Keepalive Idle TimeoutSet maximum time allowed for an idle Keepalive connection.
Max Connect TimeoutSet maximum time allowed to establish a connection, this time includes the TLS handshake.
Max Connect Timeout Log ErrorOn connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message.
Max Keepalive RecycleSet maximum number of times a keepalive connection can be used before it is retired.
Source AddressSpecify network address to bind for data traffic.