TELEMETRY PIPELINE
Processing rules

Processing rules

Processing rules transform data as it passes through your telemetry pipeline. You can use a variety of rules to perform different operations on data after it leaves its source but before it reaches its destination.

Some example use cases for processing rules include:

  • Adding a new field to each log for easier debugging and troubleshooting.
  • Redacting sensitive values to preserve user privacy.
  • Removing unnecessary fields to improve your data's signal-to-noise ratio.
  • Converting data from one format to another.
  • Turning unstructured data into structured data.
  • Aggregating logs into metrics to reduce data volume while retaining key insights.

How processing rules work

Each built-in processing rule transforms data in a specific way. By using more than one processing rule together, you can create a complex sequence of transformations that's suited for your telemetry data and storage format.

Most processing rules are compatible with most data formats. Processing rules are also designed to skip logs they're incompatible with rather than displaying an error message, which means you can use rules that apply only to certain chunks of data. For example, you can apply a broad processing rule to remove a certain field even if some logs in your pipeline don't contain that field.

Format

Processing rules are run one at a time, from top to bottom. If you add multiple processing rules to the same pipeline for the same telemetry type, the output from your first rule becomes the input for your second rule, the output from your second rule becomes the input for your third rule, and so on.

Telemetry types

Requires Core Operator version 3.1.0 or later and pipeline version 24.7.3 or later.

Processing rules support logs, metrics, and traces. You can create processing rules for multiple telemetry types within the same pipeline, but each processing rule is applied only to its specified telemetry type. For example, if you create a search/replace value processing rule for metrics, this rule won't affect any logs or traces that pass through your pipeline, even if those logs or traces contain a matching key.

When raw log data passes through at least one processing rule, the data receives a new log field for each event. This log field lets you treat each event as a single unit of data.

Structured log data such as JSON doesn't receive a log field because you can already break structured data into discrete events.

Regex engines

Requires Telemetry Pipeline version 2.9.0 or later.

Some processing rules, like Block records and Rename keys, accept regular expressions. For most of these rules, you can specify one of the following regex engines to parse your rule:

Record accessor syntax

If your raw data is in JSON format, you can use record accessor syntax to extract nested fields within a larger JSON object.

To extract a nested field inside a standard object, use the following syntax:

$objectName.fieldName

To extract a nested field inside an array, use the following syntax:

$objectName.arrayName[X]

For example, given the following JSON object:

    "log": "1234",
    "kubernetes": {
        "pod_name": "mypod-0",
        "labels": [
            {
                "app": "myapp"
            }
        ],
    }
}

The expression $kubernetes.pod_name resolves to mypod-0, and the expression $kubernetes.labels[0] resolves to "app": "myapp".

If the name of a field or parent object contains periods, wrap the name in quotes. For example, to access the field k8s.pod.uid inside the object attributes, use the expression $attributes."k8s.pod.uid".

Processing rules playground

The Telemetry Pipeline web interface has a processing rules playground (opens in a new tab) you can use to test and troubleshoot processing rules, including custom Lua scripts.

For safety reasons, this playground environment is isolated from the internet and has no access to internal Telemetry Pipeline resources. Any processing rules you test here won't affect your pipeline, clusters, or logging data.

Templates

Chronosphere offers several processing rule templates for different types of telemetry data in the processing-templates GitHub repository (opens in a new tab). You can also create your own templates from custom processing rules by using the Export feature in the processing rules interface.

To import an existing template, click Import to select a template JSON file.

Add processing rules to your pipeline

  1. Sign in to the Telemetry Pipeline web interface (opens in a new tab).

  2. Navigate to Core Instances, then select the pipeline to which you'd like to add a new processing rule.

  3. Click Edit.

  4. Click the node in the middle of the configuration diagram.

  5. In the dialog that appears, select an option from the Telemetry type tab.

  6. Click Add new action to open the processing rules menu.

  7. Select a processing rule from the available list.

  8. Configure the available settings for that processing rule, and then click Apply.

  9. Optional: Repeat steps 5 through 8 to add additional processing rules. If you add multiple rules, you can drag them to change the order in which they run.

  10. Optional: Add test input and then click Run actions to preview the output of your processing rules.

  11. Click Apply processing rules to finalize your processing rules, and then click Save and deploy to save your pipeline settings.

Use the toggle next to a processing rule to enable or disable that rule.

Available processing rules

Telemetry Pipeline offers the following processing rules:

Processing ruleDescription
Add/set key/valueAdds the same key/value pair to every record.
Aggregate recordsTransforms incoming logs into computed metrics at periodic intervals.
Allow keysPreserves any keys that match a specified regular expression, and removes all other keys.
Allow recordsPreserves any records that contain a key whose value matches a specified regular expression, and removes all other records.
Block keysRemoves keys that match a regular expression.
Block recordsRemoves records that contain a key whose value matches a specified regular expression.
Copy keysCopies the value of a specified source key to the value of a specified destination key.
Custom LuaWrite custom Lua scripts to transform your telemetry data.
Decode CSVTransforms log data from CSV format to JSON.
Decode JSONTransforms an escaped JSON string into a structured JSON object.
Deduplicate recordsSearches for any records that contain identical key/value data within a specified time frame, then removes all but the earliest of those records.
Delete keyDeletes a specified key and its associated value from all records.
Encode CSVTransforms log data from JSON to CSV format.
Encode JSONTransforms a JSON object into an escaped string.
Extract keys/valuesUses a regular expression to search for key/value pairs inside a string, then creates a structured object to store those key/value pairs.
Flatten subrecordUses regular expression to search for key/value pairs inside a JSON object, then either moves or copies any applicable key/value pairs to the top level of the record.
Hash keyCopies the value of a specified source key, hashes that value, then stores the hashed copy in a specified destination key.
Join recordsCombined values from multiple records into an array of values within a single record.
Lift submapUses regular expressions to search for key/value pairs inside a JSON object, then either moves or copies any applicable key/value pairs out of the JSON object and into a higher level of the record.
Multiline joinCombines multiple logs into a single log by looking for repeating patterns in log data.
Nest keysMoves the value of a specified source key into an object nested under a specified destination key.
ParseUses a regular expression to search for values inside a string and assign a key to each value, then stores those key/value pairs in a structured object.
Parse numberUses a regular expression to transform a key's value from a string to a number.
Random samplingPreserves a percentage of records that pass through your pipeline and discards the rest.
Redact/mask valueObscures all or part of a specified key's value by replacing the original string with a series of repeated characters.
Rename keysChanges the name of a specified key.
Search/replace valueUses regular expressions to search for a value inside a string, then replaces that value with a different specified value.
Split recordSplits an array of JSON objects into a series of standalone records.