Amazon S3 destination plugin

Amazon S3 is a highly scalable and durable object storage service provided by Amazon Web Services (AWS). The S3 destination plugin in Calyptia Core allows you to store and archive your data by sending it directly to your Amazon S3 bucket. With this plugin, you can configure your pipeline to store various data types such as logs, metrics, traces, and events, in your S3 bucket for long-term storage or archival purposes. The S3 destination plugin provides a flexible and customizable way to integrate your data with your S3 bucket, letting you tailor your storage and archival strategies to meet your specific needs.

Configuration parameters

The Amazon S3 destination plugin provides these configuration parameters.

General

AWS authentication

Advanced

Advanced networking

Shared credential file

Your shared credential file provides authentication credentials to the Amazon S3 destination plugin. This file must be an AWS credentials file (opens in a new tab) that includes an aws_access_key_id parameter and an aws_secret_access_key parameter. For example:

[default]
aws_access_key_id=ASIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

To reference this file in your plugin configuration, use the following syntax:

{{ files.NAME }}

Replace NAME with the name of your credentials file.

Bucket policies

To use the Amazon S3 destination plugin, you must grant the plugin write access to your S3 buckets. These bucket policies (opens in a new tab) are managed within Amazon S3, not within the plugin's own configuration.

For example, the following bucket policy lets the Amazon S3 destination plugin send data to a bucket named my-bucket:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": "s3:PutObject",
        "Resource": [
            "arn:aws:s3:::my-bucket",
            "arn:aws:s3:::my-bucket/*"
        ]
    }]
}