S3 Input (One Time) source plugin

Requires pipeline agent v25.8.1 or later, Core Operator v3.67.0 or later, and Pipeline CLI v3.66.0 or later.

The S3 Input (One Time) source plugin (name: go-s3-replay-plugin) lets you ingest data from an Amazon S3 bucket into a telemetry pipeline.

This is a pull-based source plugin.

This plugin doesn’t support duplicates of itself within the same pipeline.

Supported telemetry types

This plugin for Chronosphere Telemetry Pipeline supports these telemetry types:

Logs	Metrics	Traces

Requirements

To use the S3 Input (One Time) plugin, you must meet these requirements:

Your IAM user or IAM role must have the s3:GetObject (opens in a new tab) permission for your S3 bucket.

Configuration parameters

Use the parameters in this section to configure your plugin. The Telemetry Pipeline web interface uses the values in the Name column to describe the parameters. Items in the Key column are the YAML keys to use in pipeline configuration files.

Required

Name	Key	Description	Default
AWS Bucket Name	`aws_bucket_name`	Required. The name of the S3 bucket where your data is stored.	none
AWS Bucket Region	`aws_bucket_region`	Required. The name of the AWS region where your S3 bucket exists. For example: `us-east1`.	none
File Path	`logs`	Required. The glob format of the logs stored in your S3 bucket. For example: `/logs/\*.log`.	none

Advanced

Name	Key	Description	Default
S3 Assume Role ARN	`assume_role_arn`	The ARN of the role to assume in S3. This can be an ARN within the same account or across accounts.	none
S3 Role External ID	`role_external_id`	The external ID (opens in a new tab) of the role to assume in S3.	none
AWS S3 Endpoint	`aws_s3_endpoint`	The URL of your S3 endpoint.	none
Line Buffer Max Size	`max_line_buffer_size`	The maximum line size the plugin will read from JSON or plain text files.	`10MiB`
S3 Read Concurrency	`s3_read_concurrency`	The maximum number of concurrent S3 `GetObject` (opens in a new tab) calls that this plugin will make.	The number of logical CPUs allocated to each pipeline replica.

Authentication methods

The S3 Input (One Time) plugin supports the following authentication methods:

EKS Pod Identities

To use EKS Pod Identities for authentication:

In AWS, configure EKS Pod Identities (opens in a new tab).
In Pipeline CLI, add the following flag to a create pipeline or update pipeline command:
```
calyptia {create|update} pipeline --service-account VALUE
```
Replace VALUE with the name of the Kubernetes service account associated with your Pods.

IMDS

To use IMDS for authentication:

In AWS, configure IAM roles for your EC2 instance (opens in a new tab).

IRSA

To use IRSA for authentication:

In AWS, set up IRSA for your EKS cluster (opens in a new tab).
Assign an IAM role to your Kubernetes service account (opens in a new tab).
In Pipeline CLI, add the following flag to a create pipeline or update pipeline command:
```
calyptia {create|update} pipeline --service-account VALUE
```
Replace VALUE with the name of your Kubernetes service account.

Static credentials

To use static credentials for authentication:

In Telemetry Pipeline, create create secrets that contain the values of your IAM access keys (opens in a new tab). These secrets must use the key names AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

You don’t need to add an explicit reference to these secrets in your pipeline configuration file. If secrets with the correct key names are present, the S3 Input (One Time) plugin automatically detects these values and uses them for authentication.

Supported data types

The S3 Input (One Time) plugin can ingest JSON objects and plain text from files stored in S3 buckets, including gzip-compressed files. Additionally, this plugin can extract and ingest compressed and uncompressed files from tar archives.

JSON

This plugin can ingest data from JSON files with these file extensions:

.json
.jsonl
.ndjson

If a file contains only a single JSON object, this plugin creates a new log from that object. If a file contains multiple newline-delimited JSON (NDJSON) (opens in a new tab) objects, this plugin creates a new log from each JSON object within that file. Key/value pairs from JSON objects are stored as key/value pairs in the resulting log.

For JSON files that use gzip compression (with file extensions such as .json.gzip or .json.gz), this plugin decompresses each file before processing it accordingly.

Plain text

If a file doesn’t use a file extension that identifies it as a JSON file, the S3 Input (One Time) plugin processes that file as plain text. It creates a new log from each line of the file and stores the content in a key named _raw within the resulting log.

For non-JSON files that use gzip compression (with file extensions that include the .gzip or .gz suffix), this plugin decompresses each file before processing it accordingly.

Tar archives

The plugin can extract and consume files from tar archives with these file extensions:

.tar
.tar.gz
.tar.gzip

After the plugin extracts these files, it processes any JSON and plain text data accordingly, but skips directories and symbolic links.

If files inside a tar archive are gzip-compressed, this plugin decompresses those files accordingly.

Metadata

The S3 Input (One Time) plugin attaches the following metadata to each log:

__chrono_bucket: The name of the S3 bucket that contains the file from which the log was created.
__chrono_file: The key of the S3 object from which the log was created.
__chrono_tar_file_entry: For data extracted from tar archives only. The name of the tar archive that contained the file from which the log was created.

Get started

To get started with the S3 Input (One Time) plugin, follow these steps.

Either create a new pipeline (opens in a new tab) or modify an existing pipeline (opens in a new tab).
For testing purposes, set the pipeline’s destination to standard output (opens in a new tab).
Set the pipeline’s source to S3 Input (One Time), and then add values for all required parameters, along with any optional parameters of your choosing.
Set up one of the supported authentication methods for the S3 Input (One Time) source plugin.
In the Telemetry Pipeline web interface, go to the summary page for that pipeline.
In the Pipeline Output section, click Get latest logs.
Review this log output to ensure that you’re receiving data from S3. If you don’t receive any data, or if you encounter connection errors, review your plugin configuration settings.
After you’ve confirmed that the S3 Input (One Time) plugin is functioning correctly, you can overwrite the standard output destination with the destination where you want to send your telemetry data.

Prometheus Scrape S3 Input (SQS)