Skip to main content
The flatten subrecord processing rule uses regular expressions to search for key/value pairs inside a JSON object, then either moves or copies any applicable key/value pairs to the top level of the record.

​
Configuration parameters

Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the Name column to describe these parameters. Pipeline configuration files use the items in the Key column as YAML keys.
NameKeyDescriptionDefault
KeykeyRequired. The key of the object that contains key/value pairs to move or copy. You can also use record accessor syntax to reference keys nested in another nested object.none
RegexregexRequired. The regular expression that determines which key/value pairs to move or copy. A value of ^.+$ matches all key/value pairs contained by the specified object.^.+$
Replacement keykeyReplacementRequired. The regular expression that sets the naming pattern for keys that are moved or copied. A value of %1 preserves all keys’ original names.%1
Keep original checkboxkeepOrigIf selected, preserves the object specified in Key. If unselected, the flatten subrecord rule moves any applicable key/value pairs to the top level of the record, and then removes the object specified in Key.Not selected
CommentcommentA custom note or description of the rule’s function. This text is displayed next to the rule’s name in the Actions list in the processing rules interface.none

​
Examples

Using the flatten subrecord rule lets you restructure your telemetry data by pulling key/value pairs out of a nested object and storing them as flat data at the record’s root.

​
Move key/value pairs

For example, given this sample log data: 
{"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"account":{"vip":"no","id":3},"device":"mobile"},"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"account":{"vip":"yes","id":4},"device":"desktop"},"page_id":10,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":50,"action":"click"}
{"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"account":{"vip":"yes","id":5},"device":"mobile"},"page_id":40,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"account":{"vip":"no","id":1},"device":"other"},"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"account":{"vip":"yes","id":2},"device":"mobile"},"page_id":40,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"account":{"vip":"no","id":3},"device":"desktop"},"page_id":30,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":20,"action":"view"}
{"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"account":{"vip":"yes","id":2},"device":"desktop"},"page_id":50,"action":"purchase"}
{"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"account":{"vip":"yes","id":2},"device":"other"},"page_id":10,"action":"view"}
{"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"account":{"vip":"no","id":1},"device":"mobile"},"page_id":50,"action":"view"}
A processing rule with the Key value $user.account, the Regex value id, and the Replacement key value user.%1 returns the following result: 
{"user.id":3,"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"device":"mobile"},"action":"purchase","page_id":30}
{"user.id":4,"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"device":"desktop"},"action":"purchase","page_id":10}
{"user.id":1,"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"device":"desktop"},"action":"click","page_id":50}
{"user.id":5,"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"device":"mobile"},"action":"purchase","page_id":40}
{"user.id":1,"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"device":"other"},"action":"purchase","page_id":30}
{"user.id":2,"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"device":"mobile"},"action":"click","page_id":40}
{"user.id":3,"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"device":"desktop"},"action":"click","page_id":30}
{"user.id":1,"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"device":"desktop"},"action":"view","page_id":20}
{"user.id":2,"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"device":"desktop"},"action":"purchase","page_id":50}
{"user.id":2,"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"device":"other"},"action":"view","page_id":10}
{"user.id":1,"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"device":"mobile"},"action":"view","page_id":50}
This rule searched for applicable key/value pairs inside the account object, moved id up into a new key named user.id at the record’s top level, and then removed account and the key/value pairs it originally contained.

​
Copy key/value pairs

You can also use the Keep original setting to copy key/value pairs out of an object instead of moving them. For example, given this sample log data: 
{"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"account":{"vip":"no","id":3},"device":"mobile"},"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"account":{"vip":"yes","id":4},"device":"desktop"},"page_id":10,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":50,"action":"click"}
{"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"account":{"vip":"yes","id":5},"device":"mobile"},"page_id":40,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"account":{"vip":"no","id":1},"device":"other"},"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"account":{"vip":"yes","id":2},"device":"mobile"},"page_id":40,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"account":{"vip":"no","id":3},"device":"desktop"},"page_id":30,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"account":{"vip":"no","id":1},"device":"desktop"},"page_id":20,"action":"view"}
{"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"account":{"vip":"yes","id":2},"device":"desktop"},"page_id":50,"action":"purchase"}
{"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"account":{"vip":"yes","id":2},"device":"other"},"page_id":10,"action":"view"}
{"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"account":{"vip":"no","id":1},"device":"mobile"},"page_id":50,"action":"view"}
A processing rule with the Key value $user.account, the Regex value id, the Replacement key value user.%1, and the Keep original setting enabled returns the following result: 
{"user.id":3,"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"device":"mobile","account":{"vip":"no","id":3}},"action":"purchase","page_id":30}
{"user.id":4,"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"device":"desktop","account":{"vip":"yes","id":4}},"action":"purchase","page_id":10}
{"user.id":1,"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"device":"desktop","account":{"vip":"no","id":1}},"action":"click","page_id":50}
{"user.id":5,"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"device":"mobile","account":{"vip":"yes","id":5}},"action":"purchase","page_id":40}
{"user.id":1,"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"device":"other","account":{"vip":"no","id":1}},"action":"purchase","page_id":30}
{"user.id":2,"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"device":"mobile","account":{"vip":"yes","id":2}},"action":"click","page_id":40}
{"user.id":3,"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"device":"desktop","account":{"vip":"no","id":3}},"action":"click","page_id":30}
{"user.id":1,"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"device":"desktop","account":{"vip":"no","id":1}},"action":"view","page_id":20}
{"user.id":2,"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"device":"desktop","account":{"vip":"yes","id":2}},"action":"purchase","page_id":50}
{"user.id":2,"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"device":"other","account":{"vip":"yes","id":2}},"action":"view","page_id":10}
{"user.id":1,"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"device":"mobile","account":{"vip":"no","id":1}},"action":"view","page_id":50}
This rule searched for applicable key/value pairs inside the account object, copied id into a new key named user.id at the record’s top level, but did not remove account and the key/value pairs it originally contained. For a processing rule with a similar effect, see lift submap. For a processing rule with the opposite effect, see nest keys.