TELEMETRY PIPELINE
Split record

Split record

The split record processing rule splits an array of JSON objects into a series of standalone records.

For a processing rule with the opposite effect, see join records.

Configuration parameters

Use the parameters in this section to configure this processing rule. The Telemetry Pipeline web interface uses the items in the Name column to describe these parameters. Pipeline configuration files use the items in the Key column as YAML keys.

NameKeyDescriptionDefault
Source keykeyRequired. The key that contains the array to split.none
Preserve non-array data checkboxskipMissingIf selected, the split record rule preserves instances of Source key that contain non-array data. If unselected, this rule removes instances of Source key that contain non-array data.Not selected
Preserve empty arrays checkboxskipEmptyIf selected, this rule preserves instances of Source key that contain an empty array. If unselected, this rule removes instances of Source key that contain an empty array.Not selected
Preserve original array checkboxkeepOrigIf selected, this rule includes the full body of the original array inside each split object. If no Destination key is set or if Destination key is the same as Source key, the original array won't be preserved.Not selected
Destination keydestThe key to store the resulting split object. If unspecified, each split object is stored in a key with the same name as Source key.none
CommentcommentA custom note or description of the rule's function. This text is displayed next to the rule's name in the Actions list in the processing rules interface.none

Example

Using the split record processing rule lets you restructure your telemetry data by turning a combined array of events into multiple standalone events. For example, given these sample logs:

{"events":[
  {"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase"},
  {"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase"},
  {"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"},
  {"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase"},
  {"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase"},
  {"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"},
  {"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"},
  {"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"},
  {"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase"},
  {"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"},
  {"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"}
]}

A processing rule with the Source key value events and the Destination key value log returns the following result:

{"log":{"action":"purchase","user_id":3,"timestamp":"2023-03-28T09:08:41.64283645Z","page_id":30}}
{"log":{"action":"purchase","user_id":4,"timestamp":"2023-03-28T09:08:42.643343109Z","page_id":10}}
{"log":{"action":"click","user_id":1,"timestamp":"2023-03-28T09:08:48.643600498Z","page_id":50}}
{"log":{"action":"purchase","user_id":5,"timestamp":"2023-03-28T09:08:50.643773688Z","page_id":40}}
{"log":{"action":"purchase","user_id":1,"timestamp":"2023-03-28T09:08:51.643932272Z","page_id":30}}
{"log":{"action":"click","user_id":2,"timestamp":"2023-03-28T09:08:56.644080944Z","page_id":40}}
{"log":{"action":"click","user_id":3,"timestamp":"2023-03-28T09:09:03.64425954Z","page_id":30}}
{"log":{"action":"view","user_id":1,"timestamp":"2023-03-28T09:09:03.644317046Z","page_id":20}}
{"log":{"action":"purchase","user_id":2,"timestamp":"2023-03-28T09:09:10.64447719Z","page_id":50}}
{"log":{"action":"view","user_id":2,"timestamp":"2023-03-28T09:09:17.644810963Z","page_id":10}}
{"log":{"action":"view","user_id":1,"timestamp":"2023-03-28T09:09:20.644994805Z","page_id":50}}

This rule assigned each object in the original events array to a series of keys named log.