S3 Input (SQS) source plugin
The S3 Input (SQS) source plugin lets you retrieve data from S3 Input (SQS) and ingest it into Telemetry Pipeline.
Supported telemetry types
This plugin supports these telemetry types:
| Logs | Metrics | Traces |
|---|---|---|
Configuration parameters
Use the parameters in this section to configure your plugin. The Telemetry Pipeline web interface uses the values in the Name column to describe the parameters. Items in the Key column are the YAML keys to use in pipeline configuration files.
Required
| Name | Key | Description | Default |
|---|---|---|---|
| AWS Access Key | aws_access_key | Your AWS access key path. Required when using key-based authentication. | none |
| AWS Secret Key | aws_secret_key | Your AWS secret key path. Required when using key-based authentication. | none |
| AWS Bucket Name | aws_bucket_name | Required. The name of the AWS S3 bucket to fetch logs from. For example, logs. | none |
| AWS Bucket Region | aws_bucket_region | Required. The name of the AWS region where the bucket is. For example: us-east1. | none |
| AWS SQS Queue Name | sqs_queue_name | Required. The name of the queue in SQS to use. For example, testqueue. | none |
| AWS SQS Queue Region | sqs_queue_region | Required. The name of the region. For example, us-east-1. | none |
| Regular Expression Object Match | match_regexp | Required. The regular expression used to match or exclude objects from S3 from SQS queue. | .* |
| Delete Message from SQS | delete_messages | Required. Whether to delete the message from the SQS queue after processing. Accepted values: true, false. | true |
| Line Buffer Max Size | max_line_buffer_size | Required. The maximum buffer size. | 10MiB |
Advanced
| Name | Key | Description | Default |
|---|---|---|---|
| AWS S3 Endpoint | aws_s3_endpoint | The URL of the S3 endpoint. | none |
| AWS S3 Role ARN | aws_s3_role_arn | The ARN of the assumed role for S3. | none |
| AWS S3 Role Session Name | aws_s3_role_session_name | The session name of the assumed role for S3. | none |
| AWS S3 Role External ID | aws_s3_role_external_id | The external ID of the assumed role for S3. | none |
| AWS S3 Role Duration | aws_s3_role_duration | The duration of the assumed role for S3. | none |
| Enable IMDS for S3 | aws_s3_enable_imds | Indicates whether to enable IMDS authentication for S3. Accepted values: true, false. | false |
| AWS SQS Endpoint | aws_sqs_endpoint | The URL of the SQS endpoint for S3. | none |
| AWS SQS Role ARN | aws_sqs_role_arn | The ARN of the assumed role for SQS. | none |
| AWS SQS Role Session Name | aws_sqs_role_session_name | The session name of the assumed role for SQS. | none |
| AWS SQS Role External ID | aws_sqs_role_external_id | The external ID of the assumed role for SQS. | none |
| AWS SQS Role Duration | aws_sqs_role_duration | The duration of the assumed role for SQS. | none |
| Enable IMDS for SQS | aws_sqs_enable_imds | Indicates whether to enable IMDS authentication for SQS . Accepted values: true, false. | false |
| S3 Read Concurrency | s3_read_concurrency | The number of CPUs to dedicate to concurrently reading S3 data. If unspecified, the plugin uses the number of cores on the instance where it's deployed. | none |
Authentication methods
The S3 Input (SQS) plugin supports two different authentication methods. You must use one or the other, but don't need to use both at the same time.
AWS keys
To use key-based authentication, set the AWS Access Key and AWS Secret Key parameters to the values of your respective AWS keys.
IMDS
If the Core Instance where your pipeline is deployed is in AWS EC2 or AWS EKS, you can authenticate by using Instance Metadata Service (IMDS) (opens in a new tab). To use this method, you must use IMDSv2 in AWS and set values for the following advanced parameters:
- AWS S3 Endpoint
- AWS S3 Role ARN
- AWS S3 Role Session Name
- AWS S3 Role External ID
- AWS S3 Role Duration
- Enable IMDS for S3 (value:
true) - AWS SQS Endpoint
- AWS SQS Role ARN
- AWS SQS Role Session Name
- AWS SQS Role External ID
- AWS SQS Role Duration
- Enable IMDS for SQS (value:
true)
The values of the AWS S3 Role ARN and AWS SQS Role ARN parameters should
use the format arn:aws:iam::YOURIAM.
Supported data types
The S3 Input (SQS) plugin can consume JSON and gzip data from an S3 bucket. When this plugin pulls data into a pipeline, it performs the following transformations on that data:
- Unzips the data, if applicable.
- For each record, nests all keys within a top-level key named
_raw. - Inside the
_rawkey, adds keys fordate,bucket, andfilemetadata.
Get started
Follow these steps to get started with the S3 Input (SQS) source plugin.
Requirements
To use the S3 Input (SQS) plugin, you must meet all of these requirements:
- You must have an existing Amazon S3 bucket.
- You must be using Amazon SQS.
- In AWS, you must have an EC2 instance or EKS cluster with the proper IAM role assigned to it.
Set up the S3 Input (SQS) plugin
-
Either create a new pipeline or modify an existing pipeline.
-
For testing purposes, set the pipeline's destination to standard output.
-
Set the pipeline's source to S3 Input (SQS), and then add values for all required parameters, plus the parameters necessary for your authentication method of your choice.
-
Go to the summary page for that pipeline. Under Pipeline Output, click Get latest logs.
-
Review this log output to ensure that you're receiving data from S3. If you're not receiving any data, or encounter connection errors, review your plugin configuration settings.
-
After you've confirmed that the S3 Input (SQS) plugin is functioning correctly, you can overwrite the standard output destination with the destination where you'd like to send your S3 data.