S3 Input (SQS) source plugin
The S3 Input (SQS) source plugin lets you retrieve data from S3 Input (SQS) and ingest it into Telemetry Pipeline.
Supported telemetry types
This plugin supports these telemetry types:
| Logs | Metrics | Traces |
|---|---|---|
Configuration parameters
Use the parameters in this section to configure your plugin. The Telemetry Pipeline web interface uses the values in the Name column to describe the parameters. Items in the Key column are the YAML keys to use in pipeline configuration files.
Required
| Name | Key | Description | Default |
|---|---|---|---|
| AWS Access Key | aws_access_key | Required. Your AWS access key path. | none |
| AWS Secret Key | aws_secret_key | Required. Your AWS secret key path. | none |
| AWS Bucket Name | aws_bucket_name | Required. The name of the AWS S3 bucket to fetch logs from. For example, logs. | none |
| AWS Bucket Region | aws_bucket_region | Required. The name of the AWS region where the bucket is. For example: us-east1. | none |
| AWS SQS Queue Name | sqs_queue_name | Required. The name of the queue in SQS to use. For example, testqueue. | none |
| AWS SQS Queue Region | sqs_queue_region | Required. The name of the region. For example, us-east-1. | none |
| Regular Expression Object Match | match_regexp | Required. The regular expression used to match or exclude objects from S3 from SQS queue. | none |
| Delete Message from SQS | delete_messages | Required. Whether to delete the message from the SQS queue after processing. Accepted values: true, false. | false |
| Line Buffer Max Size | max_line_buffer_size | Required. The maximum buffer size. | 10MiB |
Advanced
| Name | Key | Description | Default |
|---|---|---|---|
| AWS S3 Endpoint | aws_s3_endpoint | The URL of the S3 endpoint. | none |
| AWS S3 Role ARN | aws_s3_role_arn | The ARN of the assumed role for S3. | none |
| AWS S3 Role Session Name | aws_s3_role_session_name | The session name of the assumed role for S3. | none |
| AWS S3 Role External ID | aws_s3_role_external_id | The external ID of the assumed role for S3. | none |
| AWS S3 Role Duration | aws_s3_role_duration | The duration of the assumed role for S3. | none |
| Enable IMDS for S3 | aws_s3_enable_imds | Indicates whether to enable IMDS authentication for S3. Accepted values: true, false. | false |
| AWS SQS Endpoint | aws_sqs_endpoint | The URL of the SQS endpoint for S3. | none |
| AWS SQS Role ARN | aws_sqs_role_arn | The ARN of the assumed role for SQS. | none |
| AWS SQS Role Session Name | aws_sqs_role_session_name | The session name of the assumed role for SQS. | none |
| AWS SQS Role External ID | aws_sqs_role_external_id | The external ID of the assumed role for SQS. | none |
| AWS SQS Role Duration | aws_sqs_role_duration | The duration of the assumed role for SQS. | none |
| Enable IMDS for SQS | aws_sqs_enable_imds | Indicates whether to enable IMDS authentication for SQS . Accepted values: true, false. | false |
| S3 Read Concurrency | s3_read_concurrency | The number of CPUs to dedicate to concurently reading S3 data. If unspecified, the plugin uses the number of cores on the instance where it's deployed. | none |