Processing rules - Chronosphere Documentation

This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform.

Processing rules transform the data that passes through your telemetry pipeline. You can use a variety of rules to perform different operations on data in transit between its source and its destination. Some example use cases for processing rules include:

Adding a new field to each log for easier debugging and troubleshooting.
Redacting sensitive values to preserve user privacy.
Removing unnecessary fields to improve the signal-to-noise ratio of your data.
Converting data from one format to another.
Turning unstructured data into structured data.
Aggregating logs into metrics to reduce data volume while retaining key insights.

Overview

Each built-in processing rule transforms data in a specific way. You can combine multiple processing rules to create complex transformation logic suited for your telemetry data and storage format. Processing rules are applied at the plugin level, and each plugin can have multiple associated processing rules. Any rules associated with a source plugin are applied after the data from that plugin has been parsed by a file-based parser, if applicable. Any rules associated with a destination plugin are applied before the data leaves your pipeline.

Format

Processing rules are run one at a time, from top to bottom. If you add multiple processing rules to the same pipeline for the same telemetry type, the output from your first rule becomes the input for your second rule, the output from your second rule becomes the input for your third rule, and so on. Most processing rules are compatible with most data formats. Processing rules are also designed to skip any incompatible records or records that can’t be processed, which means you can create rules even if they only apply to a subset of your data. For example, you can apply a broad rule to remove a certain field from your logs even if some logs that travel through your pipeline don’t contain that field.

Telemetry types

Requires Core Operator version 3.1.0 or later and pipeline agent version 24.7.3 or later.

Processing rules support logs, metrics, and traces. You can create processing rules for multiple telemetry types within the same pipeline, but each processing rule is applied only to its specified telemetry type. For example, if you create a search/replace value processing rule for metrics, this rule won’t affect any logs or traces that pass through your pipeline, even if those logs or traces contain a matching key.

When raw log data passes through at least one processing rule, the data receives a new log field for each event. This log field lets you treat each event as a single unit of data.Structured log data such as JSON doesn’t receive a log field because you can already break structured data into discrete events.

Regex engines

Requires Telemetry Pipeline version 2.9.0 or later.

Some processing rules, like block records and rename keys, accept regular expressions. For most of these rules, you can specify one of the following engines to parse your rule:

PCRE2 (default)
Oniguruma
POSIX
GNU
TRE

Record accessor syntax

If your raw data is in JSON format, you can use record accessor syntax to extract nested fields within a larger JSON object. To extract a nested field inside a standard object, use the following syntax:

$objectName.fieldName

To extract a nested field inside an array, use the following syntax:

$objectName.arrayName[X]

For example, given the following JSON object:

{
  "log": "1234",
  "kubernetes": {
    "pod_name": "mypod-0",
    "labels": [
      {
        "app": "myapp",
        "env": "prod"
      }
    ]
  }
}

The expression $kubernetes.pod_name resolves to mypod-0, and the expression $kubernetes.labels[0] resolves to "app": "myapp".

If the name of a field or parent object contains periods, wrap the name in quotes. For example, to access the field k8s.pod.uid inside the object attributes, use the expression $attributes."k8s.pod.uid".

Add processing rules to your pipeline

Sign in to the Telemetry Pipeline web interface.
Open the project that contains the pipeline to which you want to add processing rules.
Navigate to Core Instances, then select the pipeline to which you want to add processing rules.
Click Edit.
Click the node in the middle of the configuration diagram.
In the dialog that appears, select an option from the Telemetry type tab.
Click Add New Rule to open the processing rules menu.
Select a processing rule from the available list.
Configure the available settings for that processing rule, and then click Apply.
Optional: Repeat steps 5 through 8 to add additional processing rules. If you add multiple rules, you can drag them to change the order in which they run.
Optional: Add test input and then click Preview Output to preview the output of your processing rules.
Click Apply Processing Rules to finalize your processing rules, and then click Save and Deploy to save your pipeline settings.

Use the toggle next to a processing rule to enable or disable that rule.

Use pipeline logs as preview input data

You can use the Input and Output panes in the processing rules builder to preview the effects of your processing rules. When you edit processing rules for most pipelines, or when you use the processing rules playground, the Input pane displays sample logs by default. However, when you edit certain eligible pipelines, the Input pane automatically fetches the latest logs from that pipeline. You can also manually fetch logs from eligible pipelines to use as preview input data in the processing rules playground or when editing any other pipelines.

To be an eligible preview input source, a pipeline must be active and must include the Standard output destination plugin. This plugin is necessary because Chronosphere can’t access the telemetry data that passes through your pipelines without it.

To use a pipeline’s logs as preview input data, choose an eligible pipeline from the Select pipeline drop-down menu, then click Get Latest Logs. This takes a snapshot of the logs currently passing through your pipeline and displays that snapshot in the Input pane. If you click Preview Output, the Output pane will show how the logs in that snapshot would change based on the processing rules that you’ve configured. However, processing rules won’t actually modify the previewed logs unless you save and apply changes to the pipeline from which those logs originate. To use sample logs as preview input, open the Select pipeline drop-down menu and click Sample logs. You can also overwrite the logs in the Input pane with any text of your choosing.

Save and load templates

To create a template from your current processing rule settings, click Export to save a JSON template file to your device. You can also click Save to save your template within the Telemetry Pipeline web interface, which makes that template available to any other users in your current project. To load an existing template, click Import and select a JSON template file stored on your device, or use the Select Template drop-down menu to choose one of the templates available in your current project. Chronosphere also offers several processing rule templates in the processing-templates GitHub repository.

Available processing rules

Telemetry Pipeline offers the following processing rules:

Processing rule	Description
Add/set key/value	Adds the same key/value pair to every record.
Aggregate records	Transforms incoming logs into computed metrics at periodic intervals.
Allow keys	Preserves any keys that match a specified regular expression, and removes all other keys.
Allow records	Preserves any records that contain a key whose value matches a specified regular expression, and removes all other records.
Block keys	Removes keys that match a regular expression.
Block records	Removes records that contain a key whose value matches a specified regular expression.
Copy keys	Copies the value of a specified source key to the value of a specified destination key.
Custom Lua	Write custom Lua scripts to transform your telemetry data.
Decode CSV	Transforms log data from CSV format to JSON.
Decode JSON	Transforms an escaped JSON string into a structured JSON object.
Deduplicate records	Searches for any records that contain identical key/value data within a specified time frame, then removes all but the earliest of those records.
Delete key	Deletes a specified key and its associated value from all records.
Encode CSV	Transforms log data from JSON to CSV format.
Encode JSON	Transforms a JSON object into an escaped string.
Extract keys/values	Uses a regular expression to search for key/value pairs inside a string, then creates a structured object to store those key/value pairs.
Flatten subrecord	Uses regular expression to search for key/value pairs inside a JSON object, then either moves or copies any applicable key/value pairs to the top level of the record.
Hash key	Copies the value of a specified source key, hashes that value, then stores the hashed copy in a specified destination key.
Join records	Combined values from multiple records into an array of values within a single record.
Lift submap	Uses regular expressions to search for key/value pairs inside a JSON object, then either moves or copies any applicable key/value pairs out of the JSON object and into a higher level of the record.
Multiline join	Combines multiple logs into a single log by looking for repeating patterns in log data.
Nest keys	Moves the value of a specified source key into an object nested under a specified destination key.
Parse	Uses a regular expression to search for values inside a string and assign a key to each value, then stores those key/value pairs in a structured object.
Parse number	Uses a regular expression to transform a key’s value from a string to a number.
Random sampling	Preserves a percentage of records that pass through your pipeline and discards the rest.
Redact/mask value	Obscures all or part of a specified key’s value by replacing the original string with a series of repeated characters.
Rename keys	Changes the name of a specified key.
Search/replace value	Uses regular expressions to search for a value inside a string, then replaces that value with a different specified value.
Split record	Splits an array of JSON objects into a series of standalone records.

​Overview

​Format

​Telemetry types

​Regex engines

​Record accessor syntax

​Add processing rules to your pipeline

​Use pipeline logs as preview input data

​Save and load templates

​Available processing rules

Overview

Format

Telemetry types

Regex engines

Record accessor syntax

Add processing rules to your pipeline

Use pipeline logs as preview input data

Save and load templates

Available processing rules