Processing rules

Calyptia Core provides a set of processing rules that are built-in and ready to use. These enable you to transform your data in real time as it passes through your Telemetry pipeline.

Examples can include:

  • Adding a new field to every record
  • Blocking extraneous fields
  • Encoding or decoding CSV or other formats
  • Redacting or removing sensitive information

Processing rules use a set of actions to perform transformations on the data. These actions can work with any data type, allowing you to transform various data formats.

When you use the processing rules, you specify one or more actions that will be applied to your data as it flows through your pipeline. Each action represents a specific data transformation unit, allowing you to perform complex transformations efficiently. The collection of actions you choose for your pipeline will inform the processing rule for your pipeline.

This means that you can customize your pipeline to match the specific needs of your data and application, ensuring that you're getting the insights you need to make informed decisions.

You can specify processing rules for your pipeline using the drag-and-drop interface.

Regex engines

Requires Calyptia Core version v2.9.0 or later.

Some processing rules, like block record and rename keys, accept regular expressions. For most of these rules, you can specify one of the following regex engines to parse your rule:

Processing rules playground

The Calyptia Dashboard has a processing rules playground you can use to test and troubleshoot processing rules, including custom Lua scripts.

For safety reasons, this playground environment is isolated from the internet and has no access to internal Calyptia resources. Any processing rules you test here won't affect your pipeline, clusters, or logging data.

Add processing rules to your pipeline

You can add actions to your pipeline by using the user interface.

To add a processing rule to your pipeline with the user interface, select the pipeline to which you want to add a processing rule. From the pipeline overview page, click Edit in the top right corner.

Click the bare tile in the middle of your pipeline configuration from your Pipeline Configuration page.

You can start configuring your pipeline's processing rule by adding an action. To add an action, click the Add New tile at the right corner of your processing rule pop-up.

You can view a list of actions to include in your processing rule from the actions dropdown. Choose the actions you want to include and click Apply.

You can add multiple actions to your pipeline process rules, and after you're satisfied with your configuration, click Apply. Your pipeline will be updated immediately.

Use the toggle bar besides an Action to turn it on or off in your processing rule. This can be helpful when testing transformation logic.

Before applying your processing rules to your pipeline, you can test your actions by adding sample data to the Input Test box, then clicking Run. View the result of your data transformation in the Output box.

When using the input test, convert any text into JSON, or plain text using the toggle button beneath the input box. If you already have JSON, click the toggle and paste in the data as is.