Elasticsearch destination plugin

Elasticsearch is a search and analytics engine that is commonly used to store and analyze large volumes of machine-generated data, such as logs, metrics, and other telemetry data. The Elasticsearch destination plugin in Calyptia Core lets you integrate your telemetry data with Elasticsearch, enabling you to store, search, and visualize your data using Elasticsearch's powerful indexing and querying capabilities.

With the Elasticsearch destination plugin, you can configure your Calyptia Core pipeline to output your telemetry data to Elasticsearch.

This plugin provides a flexible and configurable way to transmit your data to Elasticsearch, letting you to customize the indexing settings, document mappings, and other attributes of your data to suit your specific needs.

Configuration parameters

The Elasticsearch destination plugin provides these configuration parameters.


HostIP address or hostname of the target Elasticsearch instance.
PortTCP port of the target Elasticsearch instance.
IndexIndex name.
Logstash formatEnable Logstash format compatibility.

Security and TLS

TLSEnable or disable TLS/SSL support.
TLS Certificate ValidationTurn TLS/SSL certificate validation on or off. TLS must be on for this setting to be enabled.
TLS Debug LevelSet TLS debug verbosity level. Accepts these values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), 4 (Verbose).
CA Certificate File PathAbsolute path to CA certificate file.
Certificate File PathAbsolute path to certificate file.
Private key File PathAbsolute path to private key file.
Private Key Path PasswordOptional password for tls.key_file file.
TLS SNI Hostname ExtensionHostname to be used for TLS SNI extension.


Replace DotsWhen enabled, replace field name dots with underscore, required by versions of Elasticsearch.
TypeType Name.
Suppress Type NameIf true, mapping types are removed. for v7.0.0 or later.
Buffer SizeSpecify the buffer size used to read the response from the Elasticsearch HTTP service. Use this option for debugging purposes where it's required to read full responses. Response size grows depending of the number of records inserted. To set an unlimited amount of memory set this value to false, otherwise the value must be according to the Unit Size specification.
PathElasticsearch accepts new data on HTTP query path /_bulk. But it is also possible to serve Elasticsearch behind a reverse proxy on a sub path. This option defines such path on the fluent-bit side. It adds only a path prefix in the indexing HTTP POST URI.
PipelineNewer versions of Elasticsearch allows to setup filters called pipelines. This option allows to define which pipeline the database should use. For performance reasons is strongly suggested to do parsing and filtering on Fluent Bit side, avoid pipelines.
Generate IDWhen enabled, generate _id for outgoing records. This prevents duplicate records when retrying ES.
Write OperationOperation to use to write in bulk requests.
ID KeyIf set, _id will be the value of the key from incoming record.
Replace DotsUse current time for index generation instead of message record.
Logstash PrefixWhen Logstash_Format is enabled, the Index name is composed using a prefix and the date. For example, if Logstash_Prefix is equal to mydata, your index will become mydata-YYYY.MM.DD. The last string appended belongs to the date when the data is being generated.
Logstash Prefix KeyWhen included: the value in the record that belongs to the key will be looked up and over-write the Logstash_Prefix for index generation. If the key/value is not found in the record then the Logstash_Prefix option will act as a fallback. Nested keys are supported through record accessor pattern.
Logstash Date FormatTime format (based on strftime) to generate the second part of the Index name.
Time KeyWhen Logstash_Format is enabled, each record will get a new timestamp field. The Time_Key property defines the name of that field.
Time Key FormatWhen Logstash_Format is enabled, this property defines the format of the timestamp.
Time Key NanosWhen Logstash_Format is enabled, enabling this property sends nanosecond precision timestamps.
Include Tag KeyWhen enabled, it append the Tag name to the record.
Tag KeyWhen Include_Tag_key is enabled, this property defines the key name of the tag in the message.

Basic authentication

HTTP UsernameBasic Auth Username.
HTTP PasswordBasic Auth Password. Requires HTTP_User to be set.

AWS authentication

Enable AWS AuthenticationEnable AWS Sigv4 Authentication.
AWS RegionAWS Region of your service.
AWS STS EndpointCustom endpoint for the AWS STS API, used with the AWS_Role_ARN option.
AWS IAM Assume RoleARN of an IAM role to assume (ex. for cross-account access).
AWS External IDSpecify an external ID for the STS API, can be used with the aws_role_arn parameter if your role requires an external ID.

Elastic Cloud authentication

Elastic Cloud IDElastic cloud ID of the cluster to connect to.
Elastic Cloud Authentication CredentialsElastic cloud authentication credentials.

Advanced networking

DNS ModeSelect the primary DNS connection type (TCP or UDP).
DNS ResolverSelect the primary DNS connection type (TCP or UDP).
Prefer IPv4Prioritize IPv4 DNS results when trying to establish a connection.
KeepaliveEnable or disable Keepalive support.
Keepalive Idle TimeoutSet maximum time allowed for an idle Keepalive connection.
Max Connect TimeoutSet maximum time allowed to establish a connection, this time includes the TLS handshake.
Max Connect Timeout Log ErrorOn connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message.
Max Keepalive RecycleSet maximum number of times a keepalive connection can be used before it is retired.
Source AddressSpecify network address to bind for data traffic.


Trace OutputWhen enabled, print the Elasticsearch API calls to stdout (for diag only).
Trace ErrorWhen enabled, print the Elasticsearch exception to stderr (for diag only).