OpenSearch destination plugin

The OpenSearch destination plugin lets you configure Chronosphere Telemetry Pipeline to send your data to OpenSearch.

Supported telemetry types

This plugin supports these telemetry types:

LogsMetricsTraces

Configuration parameters

Use the parameters in this section to configure your plugin. The Telemetry Pipeline web interface uses the values in the Name column to describe the parameters. Items in the Key column are the YAML keys to use in pipeline configuration files.

Required

NameKeyDescriptionDefault
HosthostIP address or hostname of the target OpenSearch instance.127.0.0.1
PortportTCP port of the target OpenSearch instance.9200
IndexindexIndex name.calyptia-core
Logstash Formatlogstash_formatEnable Logstash format compatibility. Accepted values: true, false.false

Security and TLS

NameKeyDescriptionDefault
TLStlsEnable or disable TLS/SSL support. Accepted values: true, false.false
TLS Certificate Validationtls.verifyTurn TLS/SSL certificate validation on or off. TLS must be on for this setting to be enabled.on
TLS Debug Leveltls.debugSet TLS debug verbosity level. Accepted values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), 4 (Verbose).1
CA Certificate File Pathtls.ca_fileAbsolute path to CA certificate file.none
Certificate File Pathtls.crt_fileAbsolute path to certificate file.none
Private Key File Pathtls.key_fileAbsolute path to private key file.none
Private Key Path Passwordtls.key_passwdOptional password for tls.key_file file.none
TLS SNI Hostname Extensiontls.vhostHostname to be used for TLS SNI extension.none

Advanced

NameKeyDescriptionDefault
Replace Dotsreplace_dotsWhen enabled, replace field name dots with underscore, required by versions of OpenSearch. Accepted values: true, false.false
TypetypeType Name._doc
Suppress Type Namesuppress_type_nameIf true, mapping types are removed. For v7.0.0 or later. Accepted values: true, false.true
Buffer Sizebuffer_sizeSpecify the buffer size used to read the response from the OpenSearch HTTP service. This option is useful for debugging purposes where is required to read full responses. Response size grows depending of the number of records inserted. To set an unlimited amount of memory set this value to false, otherwise the value must be according to the Unit Size specification512k
PathpathOpenSearch accepts new data on HTTP query path /_bulk. It's also possible to serve Elasticsearch behind a reverse proxy on a sub path. This option defines such path on the Telemetry Pipeline side. It adds only a path prefix in the indexing HTTP POST URI.none
Generate IDgenerate_idWhen enabled, generate _id for outgoing records. This prevents duplicate records when retrying ES. Accepted values: true, false.false
Write Operationwrite_operationOperation to use to write in bulk requests.create
ID Keyid_keyIf set, _id will be the value of the key from incoming records.none
Replace Dotscurrent_time_indexUse the current time for index generation instead of the message record. Accepted values: true, false.false
Logstash Prefixlogstash_prefixWhen logstash_format is enabled, the index name is composed using a prefix and the date. For example, if logstash_prefix is mydata, your index becomes mydata-YYYY.MM.DD. The last string appended belongs to the date when the data is being generated.none
Logstash Prefix Keylogstash_prefix_keyWhen included, the value in the record that belongs to the key will be looked up and over-write the logstash_prefix for index generation. If the key/value isn't found in the record, the logstash_prefix option acts as a fallback. Nested keys are supported through record accessor patterns.none
Logstash Date Formatlogstash_dateformatTime format (based on strftime) to generate the second part of the index name.%Y.%m.%d
Time Keytime_keyWhen logstash_format is enabled, each record gets a new timestamp field. The time_key property defines the name of that field.@timestamp
Time Keytime_key_formatWhen logstash_format is enabled, this property defines the format of the timestamp.none
Time Key Nanostime_key_nanosWhen logstash_format is enabled, enabling this property sends nanosecond precision timestamps. Accepted values: true, false.false
Include Tag Keyinclulde_tag_keyWhen enabled, it append the tag name to the record. Accepted values: true, false.false
Tag Keytag_keyWhen inclulde_tag_key is enabled, this property defines the key name of the tag in the message.flb-key

Basic Authentication

NameKeyDescriptionDefault
HTTP Usernamehttp_userBasic auth username.none
HTTP Passwordhttp_passwdBasic auth password. Requires http_user to be set.none

AWS Authentication

NameKeyDescriptionDefault
Enable AWS Authenticationaws_authEnable AWS Sigv4 authentication. Accepted values: true, false.false
AWS Regionaws_regionAWS region of your service.none
AWS STS Endpointaws_sts_endpointCustom endpoint for the AWS STS API, used with the aws_role_arn option.none
AWS IAM Assume Roleaws_role_arnARN of an IAM role to assume. For example, for cross-account access.none
AWS External IDaws_external_idSpecify an external ID for the STS API. Can be used with the aws_role_arn parameter if your role requires an external ID.none

Advanced Networking

NameKeyDescriptionDefault
DNS Modenet.dns.modeSelect the primary DNS connection type, which can be TCP or UDP.none
DNS Resolvernet.dns.resolverSelect the primary DNS connection type, which can be LEGACY or ASYNC.none
Prefer IPv4net.dns.prefer_ipv4Prioritize IPv4 DNS results when trying to establish a connection. Accepted values: true, false.false
Keepalivenet.keepaliveEnable or disable Keepalive support. Accepted values: true, false.true
Keepalive Idle Timeoutnet.keepalive_idle_timeoutSet maximum time allowed for an idle Keepalive connection.30s
Max Connect Timeoutnet.connect_timeoutSet maximum time allowed to establish a connection, which includes the TLS handshake.10s
Max Connect Timeout Log Errornet.connect_timeout_log_errorOn connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. Accepted values: true, false.true
Max Keepalive Recyclenet.keepalive_max_recycleSet maximum number of times a keepalive connection can be used before it's retired.2000
Source Addressnet.source_addressSpecify network address to bind for data traffic.none

Debugging

NameKeyDescriptionDefault
Trace Outputtrace_outputWhen enabled, print the OpenSearch API calls to stdout. For diagnostics only. Accepted values: true, false.false
Trace Errortrace_errorWhen enabled, print the OpenSearch exceptions to stderr. For diagnostics only. Accepted values: true, false.false