> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Amazon S3 destination plugin
export const entity_0 = "Amazon S3 destination plugin"
export const plugin_0 = "Amazon S3 destination plugin"
The Amazon S3 [destination plugin](/ingest/pipeline/plugins/destination-plugins)
(name: `s3`) lets you configure your telemetry pipeline to store and archive
your data by sending it directly to your Amazon S3 bucket.
## Supported telemetry types
The {plugin_0} for Chronosphere Telemetry Pipeline supports these telemetry types:
| Logs | Metrics | Traces |
| :----------------------------------------: | :-----------------------------: | :-----------------------------: |
| | | |
## Configuration parameters
Use the parameters in this section to configure the {entity_0}. The
Telemetry Pipeline web interface uses the items in the **Name** column to
describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files)
use the items in the **Key** column as YAML keys.
### Required
| Name | Key | Description | Default |
| --------------------------- | ----------------- | -------------------------------------------------------------------------- | ----------- |
| **Region** | `region` | Required. The AWS region to which to send your data. | `us-east-1` |
| **Bucket** | `bucket` | S3 bucket name. | *none* |
| **Total File Size (Bytes)** | `total_file_size` | Specifies the size of files in S3. Maximum size is 50 GB, minimum is 1 MB. | `1000000` |
| **Compression** | `compression` | Sets the compression type. Accepted values: `arrow`, `zip`. | *none* |
### AWS Authentication
| Name | Key | Description | Default |
| ------------------------------ | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------- |
| **AWS Shared Credential File** | `aws_shared_credentials_file` | Specifies the [shared credential file](#shared-credential-file) to use when uploading if not using AWS ARN. | *none* |
| **IAM Role ARN** | `role_arn` | Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role to assume, for uses like cross-account access. | *none* |
| **S3 Object ACL Policy** | `canned_acl` | Predefined Canned ACL policy for S3 objects. | *none* |
| **S3 API Endpoint** | `endpoint` | Custom endpoint for the AWS S3 API. | *none* |
| **STS API Endpoint** | `sts_endpoint` | Custom endpoint for the STS API. | *none* |
| **External ID for STS API** | `external_id` | Specifies an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID. | *none* |
### Advanced
| Name | Key | Description | Default |
| -------------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- |
| **Use Put Object** | `use_put_object` | Whether to use the S3 PutObject API, instead of multipart upload API. Accepted values: `true`, `false`. | `false` |
| **Send Content-MD5 header** | `send_content_md5` | Sends the Content-MD5 header with object uploads as is required when Object Lock is Enabled. Accepted values: `true`, `false`. | `false` |
| **Preserve Data Ordering** | `preserve_data_ordering` | Normally, when an upload request fails, there is a high chance for the last received chunk to be swapped with a later chunk, resulting in data shuffling. This feature prevents this shuffling by using a queue logic for uploads. Accepted values: `true`, `false`. | `false` |
| **Log Key** | `log_key` | By default, the entire log record is be sent to S3. If you specify a key name with this option, only the value of that key is sent to S3. | *none* |
| **Storage Class** | `storage_class` | Specifies the storage class for S3 objects. If this option isn't specified, objects are stored with the default `STANDARD` storage class. | *none* |
| **Store Dir** | `store_dir` | Directory to locally buffer data before sending. Plugin uses the S3 Multipart upload API to send data in chunks of 5 MB at a time. Only a small amount of data is locally buffered at any given point in time. | `/tmp/calyptia-core/s3` |
| **S3 Key Format** | `s3_key_format` | Format string for keys in S3. This option supports `strftime` time formats and a syntax for selecting parts of the Fluent log tag using a syntax inspired by the `rewrite_tag` filter. Add `$TAG` in the format string to insert the full log tag. Add `$TAG\[0]` to insert the first part of the tag in the S3 key. The tag is split into parts using the characters specified with the `s3_key_format_tag_delimiters` option. Add `$INDEX` to enable sequential indexing for file names, which prevents random string being added to end of the key when the `$UUID` isn't provided. | `/calyptia-core-logs/$TAG/%Y/%m/%d/%H/%M/%S` |
| **S3 Key Format Tag Delimiters** | `s3_key_format_tag_delimiters` | A series of characters used to split the tag into parts for use with the `s3_key_format` option. | `.` |
| **Use Static File Path?** | `static_file_path` | Disables behavior where the UUID string is automatically appended to end of the S3 key name when `$UUID` isn't provided in `s3_key_format`. `$UUID`, time formats, `$TAG`, and other dynamic key formats all work as expected when this feature is set to `true`. Accepted values: `true`, `false`. | `false` |
| **Enable Auto Retry Requests** | `auto_retry_requests` | Whether to immediately retry failed requests to AWS services one time. This option doesn't affect the normal Telemetry Pipeline retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput when there are transient or random networking issues. Accepted values: `true`, `false`. | `true` |
| **JSON Date Format** | `json_date_format` | Specifies the format of the date. Accepted values: `double`, `epoch`, `iso8601` (for example, `2018-05-30T09:39:52.000681Z`), `java_sql_timestamp` (for example, `2018-05-30 09:39:52.000681`, which can be used with AWS Athena). | *none* |
| **JSON Date Key** | `json_date_key` | Specifies the name of the date field in output. | *none* |
| **Upload Chunk Size (Bytes)** | `upload_chunk_size` | This plugin uses the S3 Multipart Upload API to stream data to S3, which helps speed up data uploads. This parameter configures the size of each `part` in the upload. The `total_file_size` option configures the size of the file in S3. This option determines the size of chunks uploaded until that size is reached. These chunks are temporarily stored in `chunk_buffer_path` until their size reaches `upload_chunk_size`, at which point the chunk is uploaded to S3. Maximum value is 50 MB, minimum is 5 MB. | `5242880` |
| **Upload Timeout** | `upload_timeout` | Optionally specifies a timeout for uploads. Whenever this amount of time has elapsed, Telemetry Pipeline completes an upload and creates a new file in S3. For example, set this value to 60m to get a new file in S3 every hour. | `10m` |
### Advanced Networking
| Name | Key | Description | Default |
| --------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| **DNS Mode** | `net.dns.mode` | Selects the primary DNS connection type, which can be `TCP` or `UDP`. | *none* |
| **DNS Resolver** | `net.dns.resolver` | Selects the primary DNS connection type, which can be `LEGACY` or `ASYNC`. | *none* |
| **Prefer IPv4** | `net.dns.prefer_ipv4` | Prioritizes IPv4 DNS results when trying to establish a connection. Accepted values: `true`, `false`. | `false` |
| **Keepalive** | `net.keepalive` | Enables or disables Keepalive support. Accepted values: `true`, `false`. | `true` |
| **Keepalive Idle Timeout** | `net.keepalive_idle_timeout` | Sets the maximum time allowed for an idle Keepalive connection. | `30s` |
| **Max Connect Timeout** | `net.connect_timeout` | Sets the maximum time allowed to establish a connection, which includes the TLS handshake. | `10s` |
| **Max Connect Timeout Log Error** | `net.connect_timeout_log_error` | Specifies whether to log an error on connection timeout. When disabled, the timeout is logged as a debug message. Accepted values: `true`, `false`. | `true` |
| **Source Address** | `net.source_address` | Specifies the network address to bind for data traffic. | *none* |
| **Max Keepalive Recycle** | `net.keepalive_max_recycle` | Sets the maximum number of times a keepalive connection can be used before it's retired. | `2000` |
## Shared credential file
Your shared credential file provides authentication credentials to the Amazon S3
destination plugin. This file must be an
[AWS credentials file](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html#cli-configure-files-format)
that includes an `aws_access_key_id` parameter and an `aws_secret_access_key`
parameter. For example:
```shell filename="Credentials" theme={null}
[default]
aws_access_key_id=ASIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
To reference this file in your plugin configuration, use the following syntax:
```yaml theme={null}
{{ files.NAME }}
```
Replace *`NAME`* with the name of your credentials file.
## Bucket policies
To use the Amazon S3 destination plugin, you must grant the plugin write access
to your S3 buckets. These
[bucket policies](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html)
are managed within Amazon S3, not within the plugin's own configuration.
For example, the following bucket policy lets the Amazon S3 destination plugin
send data to a bucket named `my-bucket`:
```json filename="Bucket policy" theme={null}
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/*"
]
}]
}
```