> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.
# S3 Input (One Time) source plugin
export const entity_0 = "S3 Input (One Time) source plugin"
export const plugin_0 = "S3 Input (One Time) source plugin"
> Requires [pipeline agent](/ingest/pipeline/v2/component-versions) v25.8.1 or later,
> [Core Operator](/ingest/pipeline/v2/component-versions) v3.67.0 or later, and
> [Pipeline CLI](/ingest/pipeline/pipeline-cli) v3.66.0 or later.
The S3 Input (One Time) [source plugin](/ingest/pipeline/plugins/source-plugins)
(name: `go-s3-replay-plugin`) lets you ingest data from an Amazon S3 bucket into
a telemetry pipeline.
This is a [pull-based](/ingest/pipeline/plugins/source-plugins#push-based-and-pull-based-source-plugins) source plugin.
This plugin doesn't support duplicates of itself within the same pipeline.
## Supported telemetry types
The {plugin_0} for Chronosphere Telemetry Pipeline supports these telemetry types:
| Logs | Metrics | Traces |
| :----------------------------------------: | :-----------------------------: | :-----------------------------: |
| | | |
## Requirements
To use the S3 Input (One Time) plugin, you must meet these requirements:
* Your IAM user or IAM role must have the
[`s3:GetObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html)
permission for your S3 bucket.
## Configuration parameters
Use the parameters in this section to configure the {entity_0}. The
Telemetry Pipeline web interface uses the items in the **Name** column to
describe these parameters. [Pipeline configuration files](/ingest/pipeline/v2/configure/config-files)
use the items in the **Key** column as YAML keys.
### Required
| Name | Key | Description | Default |
| ------------------------ | ---------------------- | -------------------------------------------------------------------------------------------------- | ------- |
| **AWS Bucket Name** | `aws_bucket_name` | Required. The name of the S3 bucket where your data is stored. | *none* |
| **AWS Bucket Region** | `aws_bucket_region` | Required. The name of the AWS region where your S3 bucket exists. For example: `us-east1`. | *none* |
| **File Path** | `logs` | Required. The glob format of the logs stored in your S3 bucket. For example: `/logs/\*.log`. | *none* |
| **Line Buffer Max Size** | `max_line_buffer_size` | The maximum line size the plugin will read from [JSON or plain text files](#supported-data-types). | `10MiB` |
### Advanced
| Name | Key | Description | Default |
| :---------------------- | :-------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------ |
| **S3 Assume Role ARN** | `assume_role_arn` | The ARN of the role to assume in S3. This can be an ARN within the same account or across accounts. | *none* |
| **S3 Role External ID** | `role_external_id` | The [external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html) of the role to assume in S3. | *none* |
| **S3 Endpoint** | `aws_s3_endpoint` | The URL of your S3 endpoint. | *none* |
| **S3 Read Concurrency** | `s3_read_concurrency` | The maximum number of concurrent S3 [`GetObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) calls that this plugin will make. | The number of logical CPUs allocated to each pipeline [replica](/ingest/pipeline/v2/configure/scaling). |
| **Memory Buffer Limit** | `mem_buf_limit` | For pipelines with the Deployment or DaemonSet [workload](/ingest/pipeline/v2/configure/kubernetes/workloads) type only. Sets a limit for how much buffered data the plugin can write to memory, which affects [backpressure](/ingest/pipeline/v2/configure/backpressure). This value must follow Fluent Bit's rules for [unit sizes](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit#unit-sizes). If unspecified, no limit is enforced. | *none* |
## Authentication methods
The S3 Input (One Time) plugin supports the following authentication methods:
* [EKS Pod Identities](#eks-pod-identities)
* [IMDS](#imds)
* [IRSA](#irsa)
* [Static credentials](#static-credentials)
### EKS Pod Identities
To use EKS Pod Identities for authentication:
1. In AWS, configure [EKS Pod Identities](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html).
2. In [Pipeline CLI](/ingest/pipeline/pipeline-cli), add the following flag to a `create pipeline`
or `update pipeline` command:
```shell /VALUE/ theme={null}
calyptia {create|update} pipeline --service-account VALUE
```
Replace *`VALUE`* with the name of the Kubernetes service account associated
with your Pods.
### IMDS
To use IMDS for authentication:
* In AWS, configure [IAM roles for your EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
### IRSA
To use IRSA for authentication:
1. In AWS, [set up IRSA for your EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html).
2. [Assign an IAM role to your Kubernetes service account](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html).
3. In [Pipeline CLI](/ingest/pipeline/pipeline-cli), add the following flag to a `create pipeline`
or `update pipeline` command:
```shell /VALUE/ theme={null}
calyptia {create|update} pipeline --service-account VALUE
```
Replace *`VALUE`* with the name of your Kubernetes service account.
### Static credentials
To use static credentials for authentication:
* In Telemetry Pipeline, create [create secrets](/ingest/pipeline/v2/configure/secrets#add-a-secret)
that contain the values of your [IAM access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
These secrets must use the key names `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
You don't need to add an explicit [reference](/ingest/pipeline/v2/configure/secrets#reference-a-secret)
to these secrets in your pipeline configuration file. If secrets with the correct
key names are present, the S3 Input (One Time) plugin automatically detects these values
and uses them for authentication.
## Supported data types
The S3 Input (One Time) plugin can ingest [JSON objects](#json) and [plain text](#plain-text)
from files stored in S3 buckets, including gzip-compressed files. Additionally,
this plugin can extract and ingest compressed and uncompressed files from
[tar archives](#tar-archives).
### JSON
This plugin can ingest data from JSON files with these file extensions:
* `.json`
* `.jsonl`
* `.ndjson`
If a file contains only a single JSON object, this plugin creates
a new log from that object. If a file contains multiple
[newline-delimited JSON (NDJSON)](https://github.com/ndjson/ndjson-spec)
objects, this plugin creates a new log from each JSON object within that file.
Key/value pairs from JSON objects are stored as key/value pairs in the resulting log.
For JSON files that use gzip compression (with file extensions such as `.json.gzip`
or `.json.gz`), this plugin decompresses each file before processing it accordingly.
### Plain text
If a file doesn't use a file extension that identifies it as a JSON file, the
S3 Input (One Time) plugin processes that file as plain text. It creates a new log from
each line of the file and stores the content in a key named `_raw` within the
resulting log.
For non-JSON files that use gzip compression (with file extensions that include
the `.gzip` or `.gz` suffix), this plugin decompresses each file before processing
it accordingly.
### Tar archives
The plugin can extract and consume files from tar archives with these file extensions:
* `.tar`
* `.tar.gz`
* `.tar.gzip`
After the plugin extracts these files, it processes
any [JSON](#json) and [plain text](#plain-text) data accordingly, but skips directories
and symbolic links.
If files inside a tar archive are gzip-compressed, this plugin decompresses those
files accordingly.
## Metadata
The S3 Input (One Time) plugin attaches the following metadata to each log:
* `__chrono_bucket`: The name of the S3 bucket that contains the file from which
the log was created.
* `__chrono_file`: The key of the S3 object from which the log was created.
* `__chrono_tar_file_entry`: For data extracted from [tar archives](#tar-archives)
only. The name of the tar archive that contained the file from which the log
was created.
## Get started
To get started with the S3 Input (One Time) plugin, follow these steps.
1. Either [create a new pipeline](/ingest/pipeline/v2/build/create-modify#create-a-pipeline)
or [modify an existing pipeline](/ingest/pipeline/v2/build/create-modify).
2. For testing purposes, set the pipeline's destination to
[standard output](/ingest/pipeline/plugins/destination-plugins/standard-output).
3. Set the pipeline's source to S3 Input (One Time), and then add values for all required
parameters, along with any optional parameters of your choosing.
4. Set up one of the supported [authentication methods](#authentication-methods)
for the S3 Input (One Time) source plugin.
5. In the Telemetry Pipeline web interface, go to the summary page for that pipeline.
6. In the [**Pipeline Output**](/ingest/pipeline/navigate#pipeline-output-v2-pipelines) section,
click **Get latest logs**.
7. Review this log output to ensure that you're receiving data from S3. If you don't
receive any data, or if you encounter connection errors, review your plugin
configuration settings.
8. After you've confirmed that the S3 Input (One Time) plugin is functioning correctly,
you can overwrite the standard output destination with the destination where
you want to send your telemetry data.