Skip to main content
The Amazon S3 destination plugin (name: s3) lets you configure your telemetry pipeline to store and archive your data by sending it directly to your Amazon S3 bucket.

​
Supported telemetry types

The for Chronosphere Telemetry Pipeline supports these telemetry types:
LogsMetricsTraces

​
Configuration parameters

Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the Name column to describe these parameters. Pipeline configuration files use the items in the Key column as YAML keys.

​
Required

NameKeyDescriptionDefault
RegionregionRequired. The AWS region to which to send your data.us-east-1
BucketbucketS3 bucket name.none
Total File Size (Bytes)total_file_sizeSpecifies the size of files in S3. Maximum size is 50 GB, minimum is 1 MB.1000000
CompressioncompressionSets the compression type. Accepted values: arrow, zip.none

​
AWS Authentication

NameKeyDescriptionDefault
AWS Shared Credential Fileaws_shared_credentials_fileSpecifies the shared credential file to use when uploading if not using AWS ARN.none
IAM Role ARNrole_arnAmazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role to assume, for uses like cross-account access.none
S3 Object ACL Policycanned_aclPredefined Canned ACL policy for S3 objects.none
S3 API EndpointendpointCustom endpoint for the AWS S3 API.none
STS API Endpointsts_endpointCustom endpoint for the STS API.none
External ID for STS APIexternal_idSpecifies an external ID for the STS API. Can be used with the role_arn parameter if your role requires an external ID.none

​
Advanced

NameKeyDescriptionDefault
Use Put Objectuse_put_objectWhether to use the S3 PutObject API, instead of multipart upload API. Accepted values: true, false.false
Send Content-MD5 headersend_content_md5Sends the Content-MD5 header with object uploads as is required when Object Lock is Enabled. Accepted values: true, false.false
Preserve Data Orderingpreserve_data_orderingNormally, when an upload request fails, there is a high chance for the last received chunk to be swapped with a later chunk, resulting in data shuffling. This feature prevents this shuffling by using a queue logic for uploads. Accepted values: true, false.false
Log Keylog_keyBy default, the entire log record is be sent to S3. If you specify a key name with this option, only the value of that key is sent to S3.none
Storage Classstorage_classSpecifies the storage class for S3 objects. If this option isn’t specified, objects are stored with the default STANDARD storage class.none
Store Dirstore_dirDirectory to locally buffer data before sending. Plugin uses the S3 Multipart upload API to send data in chunks of 5 MB at a time. Only a small amount of data is locally buffered at any given point in time./tmp/calyptia-core/s3
S3 Key Formats3_key_formatFormat string for keys in S3. This option supports strftime time formats and a syntax for selecting parts of the Fluent log tag using a syntax inspired by the rewrite_tag filter. Add $TAG in the format string to insert the full log tag. Add $TAG\[0] to insert the first part of the tag in the S3 key. The tag is split into parts using the characters specified with the s3_key_format_tag_delimiters option. Add $INDEX to enable sequential indexing for file names, which prevents random string being added to end of the key when the $UUID isn’t provided./calyptia-core-logs/$TAG/%Y/%m/%d/%H/%M/%S
S3 Key Format Tag Delimiterss3_key_format_tag_delimetersA series of characters used to split the tag into parts for use with the s3_key_format option..
Use Static File Path?static_file_pathsDisables behavior where the UUID string is automatically appended to end of the S3 key name when $UUID isn’t provided in s3_key_format. $UUID, time formats, $TAG, and other dynamic key formats all work as expected when this feature is set to true. Accepted values: true, false.false
Enable Auto Retry Requestsauto_retry_requestsWhether to immediately retry failed requests to AWS services one time. This option doesn’t affect the normal Telemetry Pipeline retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient or random networking issues. Accepted values: true, false.true
JSON Date Formatjson_date_formatSpecifies the format of the date. Accepted values: double, epoch, iso8601 (for example, 2018-05-30T09:39:52.000681Z), java_sql_timestamp (for example, 2018-05-30 09:39:52.000681, which can be used with AWS Athena).none
JSON Date Keyjson_date_keySpecifies the name of the date field in output.none
Upload Chunk Size (Bytes)upload_chunk_sizeThis plugin uses the S3 Multipart Upload API to stream data to S3, which helps speed up data uploads. This parameter configures the size of each part in the upload. The total_file_size option configures the size of the file in S3. This option determines the size of chunks uploaded until that size is reached. These chunks are temporarily stored in chunk_buffer_path until their size reaches upload_chunk_size, at which point the chunk is uploaded to S3. Maximum value is 50 MB, minimum is 5 MB.5242880
Upload Timeoutupload_timeoutOptionally specifies a timeout for uploads. Whenever this amount of time has elapsed, Telemetry Pipeline completes an upload and creates a new file in S3. For example, set this value to 60m to get a new file in S3 every hour.10m

​
Advanced Networking

NameKeyDescriptionDefault
DNS Modenet.dns.modeSelects the primary DNS connection type, which can be TCP or UDP.none
DNS Resolvernet.dns.resolverSelects the primary DNS connection type, which can be LEGACY or ASYNC.none
Prefer IPv4net.dns.prefer_ipv4Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: true, false.false
Keepalivenet.keepaliveEnables or disables Keepalive support. Accepted values: true, false.true
Keepalive Idle Timeoutnet.keepalive_idle_timeoutSets the maximum time allowed for an idle Keepalive connection.30s
Max Connect Timeoutnet.connect_timeoutSets the maximum time allowed to establish a connection, which includes the TLS handshake.10s
Max Connect Timeout Log Errornet.connect_timeout_log_errorSpecifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: true, false.true
Max Keepalive Recyclenet.keepalive_max_recycleSets the maximum number of times a keepalive connection can be used before it’s retired.2000
Source Addressnet.source_addressSpecifies the network address to bind for data traffic.none

​
Shared credential file

Your shared credential file provides authentication credentials to the Amazon S3 destination plugin. This file must be an AWS credentials file that includes an aws_access_key_id parameter and an aws_secret_access_key parameter. For example: 
[default]
aws_access_key_id=ASIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
To reference this file in your plugin configuration, use the following syntax: 
{{ files.NAME }}
Replace NAME with the name of your credentials file.

​
Bucket policies

To use the Amazon S3 destination plugin, you must grant the plugin write access to your S3 buckets. These bucket policies are managed within Amazon S3, not within the plugin’s own configuration. For example, the following bucket policy lets the Amazon S3 destination plugin send data to a bucket named my-bucket: 
{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": "s3:PutObject",
        "Resource": [
            "arn:aws:s3:::my-bucket",
            "arn:aws:s3:::my-bucket/*"
        ]
    }]
}