Parsers (Telemetry Pipeline)

TELEMETRY PIPELINE

Parsers

The telemetry data emitted by different sources often has inconsistent information and formats, which can cause problems if data from multiple sources passes through a single pipeline. It can also be difficult to analyze and process raw, unstructured data.

Parsers address these issues by transforming your telemetry data into a predictable, structured format.

The parse processing rule transforms data in a similar way.

Overview

All pipelines include a parsers.conf file with a list of default parsers for transforming common telemetry formats, including Docker, Apache, and NGINX logs. You can also create your own parsers and add them to a pipeline's parsers file.

Parsers are applied at the source plugin level. Even though a parsers.conf file can include multiple parsers, you can only apply a single parser to each source within a pipeline.

⚠️

When a parser is applied to a source plugin, parsing operations are performed on all data ingested by that plugin. Trying to ingest data that isn't compatible with the plugin's parser will cause errors.

You can also use parsers and processing rules in tandem. File-based parsers are applied to data first, and then any processing rules are applied to the resulting parsed data.

Parser syntax

The default parsers.conf file included with each pipeline is identical to the default Fluent Bit parsers.conf file (opens in a new tab). Each parser begins with the [PARSER] header and then lists its associated key/value pairs. For example, this is the default NGINX parser:

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

Custom parsers

You can add new parsers or modify existing parsers in a pipeline's parsers.conf file.

Parser settings

Parsers in Chronosphere Telemetry Pipeline use the same settings as parsers in Fluent Bit. To learn more, consult the Fluent Bit parser configuration parameters (opens in a new tab) documentation.

Update parsers file

To add a custom parser to a pipeline, you'll need to update the parsers file for that pipeline.

In Pipeline CLI, run the following command to return the contents of your pipeline's current parsers file:
```
calyptia get pipeline_file --pipeline PIPELINE --name parsers --only-contents > parsers.conf
```
Replace PIPELINE with the name of your pipeline.
Modify the contents of parsers.conf to include any custom parsers.
Run the following command to replace the original parsers file with your newly modified version:
```
calyptia update pipeline_file --pipeline PIPELINE --file ./parsers.conf
```
Replace PIPELINE with the name of your pipeline.

Apply a parser to a source plugin

To apply a parser to a source plugin, add a parser key to the YAML entry for that plugin in your pipeline's configuration file. For example, the following snippet shows a source plugin that uses a parser named custom_parser:

my-pipeline.yaml

pipeline:
  inputs:
    - Name: forward
      parser: custom_parser
      port: "5170"
      tls.verify: on
      tls.debug: "1"
<...>

The value of parser must match the Name value of one of the entries in that pipeline's parser.conf file. For example, to use the NGINX parser shown in this guide's Syntax example, set parser to nginx.

A source plugin can only use a single parser, but you can apply a parser to each source plugin within a pipeline. Parsers operate independently between plugins; any two source plugins within the same pipeline can use different parsers or can use the same parser.

Metadata Pipeline configuration files