Amazon S3

Amazon S3 destination plugin

The Amazon S3 destination plugin lets you configure Chronosphere Telemetry Pipeline to store and archive your data by sending it directly to your Amazon S3 bucket.

Configuration parameters

The Amazon S3 destination plugin provides these configuration parameters. Items in the Name column display in the Telemetry Pipeline web interface. Items in the Key column are the YAML keys to use in pipeline configuration files.

Required

Name	Key	Description	Default
Region	`region`	The AWS region of your S3 bucket.	`us-east-1`
Bucket	`bucket`	S3 bucket name.	none
Total File Size (Bytes)	`total_file_size`	Specifies the size of files in S3. Maximum size is 50 GB, minimum is 1 MB.	`1000000`
Compression	`compression`	Sets the compression type. Accepted values: `arrow`, `zip`.	none

AWS Authentication

Name	Key	Description	Default
AWS Shared Credential File	`aws_shared_credentials_file`	Specifies the shared credential file to use when uploading if not using AWS ARN.	none
IAM Role ARN	`role_arn`	ARN of an IAM role to assume. For example, for cross account access.	none
S3 Object ACL Policy	`canned_acl`	Predefined Canned ACL policy for S3 objects.	none
S3 API Endpoint	`endpoint`	Custom endpoint for the AWS S3 API.	none
STS API Endpoint	`sts_endpoint`	Custom endpoint for the STS API.	none
External ID for STS API	`external_id`	Specify an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID.	none

Advanced

Name	Key	Description	Default
Use Put Object	`use_put_object`	Use the S3 PutObject API, instead of multipart upload API.	`false`
Send Content-MD5 header	`send_content_md5`	Send Content-MD5 header with object uploads as is required when Object Lock is Enabled.	`false`
Preserve Data Ordering	`preserve_data_ordering`	Normally, when an upload request fails, there is a high chance for the last received chunk to be swapped with a later chunk, resulting in data shuffling. This feature prevents this shuffling by using a queue logic for uploads.	`false`
Log Key	`log_key`	By default, the entire log record is be sent to S3. If you specify a key name with this option, only the value of that key is sent to S3.	none
Storage Class	`storage_class`	Specify the storage class for S3 objects. If this option isn't specified, objects are stored with the default `STANDARD` storage class.	none
Store Dir	`store_dir`	Directory to locally buffer data before sending. Plugin uses the S3 Multipart upload API to send data in chunks of 5 MB at a time. Only a small amount of data is locally buffered at any given point in time.	`/tmp/calyptia-core/s3`
S3 Key Format	`s3_key_format`	Format string for keys in S3. This option supports `strftime` time formats and a syntax for selecting parts of the Fluent log tag using a syntax inspired by the `rewrite_tag` filter. Add `$TAG` in the format string to insert the full log tag. Add `$TAG\[0]` to insert the first part of the tag in the S3 key. The tag is split into parts using the characters specified with the `s3_key_format_tag_delimiters` option. Add `$INDEX` to enable sequential indexing for file names, which prevents random string being added to end of the key when the `$UUID` isn't provided.	`/calyptia-core-logs/$TAG/%Y/%m/%d/%H/%M/%S`
S3 Key Format Tag Delimiters	`s3_key_format_tag_delimeters`	A series of characters used to split the tag into parts for use with the `s3_key_format` option.	`.`
Use Static File Path?	`static_file_paths`	Disables behavior where the UUID string is automatically appended to end of the S3 key name when `$UUID` isn't provided in `s3_key_format`. `$UUID`, time formats, `$TAG`, and other dynamic key formats all work as expected when this feature is set to true.	`false`
Enable Auto Retry Requests	`auto_retry_requests`	Immediately retry failed requests to AWS services once. This option doesn't affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient or random networking issues.	`true`
JSON Date Format	`json_date_format`	Specify the format of the date. Accepted values: `double`, `iso8601` (for example, `2018-05-30T09:39:52.000681Z`), `java_sql_timestamp` (for example, `2018-05-30 09:39:52.000681`, which can be used with AWS Athena), `epoch`.	none
JSON Date Key	`json_date_key`	Specifies the name of the date field in output.	none
Upload Chunk Size (Bytes)	`upload_chunk_size`	This plugin uses the S3 Multipart Upload API to stream data to S3, which helps speed up data uploads. This parameter configures the size of each `part` in the upload. The `total_file_size` option configures the size of the file in S3. This option determines the size of chunks uploaded until that size is reached. These chunks are temporarily stored in `chunk_buffer_path` until their size reaches `upload_chunk_size`, at which point the chunk is uploaded to S3. Maximum value is 50 MB, minimum is 5 MB.	`5242880`
Upload Timeout	`upload_timeout`	Optionally specify a timeout for uploads. Whenever this amount of time has elapsed, Fluent Bit completes an upload and creates a new file in S3. For example, set this value to 60m to get a new file in S3 every hour.	`10m`

Advanced Networking

Name	Key	Description	Default
DNS Mode	`net.dns.mode`	Select the primary DNS connection type, which can be `TCP` or `UDP`.	none
DNS Resolver	`net.dns.resolver`	Select the primary DNS connection type, which can be `LEGACY` or `ASYNC`.	none
Prefer IPv4	`net.dns.prefer_ipv4`	Prioritize IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`.	`false`
Keepalive	`net.keepalive`	Enable or disable Keepalive support. Accepted values: `true`, `false`.	`true`
Keepalive Idle Timeout	`net.keepalive_idle_timeout`	Set maximum time allowed for an idle Keepalive connection.	`30s`
Max Connect Timeout	`net.connect_timeout`	Set maximum time allowed to establish a connection, which includes the TLS handshake.	`10s`
Max Connect Timeout Log Error	`net.connect_timeout_log_error`	On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`.	`true`
Max Keepalive Recycle	`net.keepalive_max_recycle`	Set maximum number of times a keepalive connection can be used before it's retired.	`2000`
Source Address	`net.source_address`	Specify network address to bind for data traffic.	none

Shared credential file

Your shared credential file provides authentication credentials to the Amazon S3 destination plugin. This file must be an AWS credentials file (opens in a new tab) that includes an aws_access_key_id parameter and an aws_secret_access_key parameter. For example:

Credentials

[default]
aws_access_key_id=ASIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

To reference this file in your plugin configuration, use the following syntax:

{{ files.NAME }}

Replace NAME with the name of your credentials file.

Bucket policies

To use the Amazon S3 destination plugin, you must grant the plugin write access to your S3 buckets. These bucket policies (opens in a new tab) are managed within Amazon S3, not within the plugin's own configuration.

For example, the following bucket policy lets the Amazon S3 destination plugin send data to a bucket named my-bucket:

Bucket policy

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": "s3:PutObject",
        "Resource": [
            "arn:aws:s3:::my-bucket",
            "arn:aws:s3:::my-bucket/*"
        ]
    }]
}

Amazon Kinesis Streams Axiom