Allow keys
The allow keys processing rule preserves any keys that match a specified regular expression, and remove all other keys.
For processing rule with the opposite effect, see block keys.
Configuration parameters
Use the parameters in this section to configure this processing rule. The Telemetry Pipeline web interface uses the items in the Name column to describe these parameters. Pipeline configuration files use the items in the Key column as YAML keys.
| Name | Key | Description | Default |
|---|---|---|---|
| Regex | regex | Required. The regular expression that determines which keys to preserve. | none |
| Match case checkbox | matchCase | Indicates whether the regular expression is case-sensitive. | Not selected |
| Regex engine | regexEngine | Required. The engine to parse your regular expression. Accepted values: GNU, Oniguruma, PCRE2, POSIX, TRE. | PCRE2 |
| Nested access pattern | nestedPath | The key of the highest object level to evaluate. If specified, the allow keys rule ignores any keys not contained within the specified object. If unspecified, this rule evaluates all top-level keys, but no keys nested within top-level keys. You can also use record accessor syntax to reference keys nested within another nested object. | none |
| Comment | comment | A custom note or description of the rule's function. This text is displayed next to the rule's name in the Actions list in the processing rules interface. | none |
Examples
Using the allow keys rule lets you pare down telemetry data by retaining only the data you explicitly choose to keep.
Flat example
For example, given this sample log data:
{"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"}
{"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"}
{"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase"}
{"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"}
{"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"}A processing rule with the Regex value user returns the following result:
{"user_id":3}
{"user_id":4}
{"user_id":1}
{"user_id":5}
{"user_id":1}
{"user_id":2}
{"user_id":3}
{"user_id":1}
{"user_id":2}
{"user_id":2}
{"user_id":1}Nested example
You can also use the allow keys rule to selectively retain information within a nested object. For example, given this sample log data:
{"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"vip":"no","id":3},"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"vip":"yes","id":4},"page_id":10,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"vip":"no","id":1},"page_id":50,"action":"click"}
{"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"vip":"yes","id":5},"page_id":40,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"vip":"no","id":1},"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"vip":"yes","id":2},"page_id":40,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"vip":"no","id":3},"page_id":30,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"vip":"no","id":1},"page_id":20,"action":"view"}
{"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"vip":"yes","id":2},"page_id":50,"action":"purchase"}
{"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"vip":"yes","id":2},"page_id":10,"action":"view"}
{"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"vip":"no","id":1},"page_id":50,"action":"view"}A processing rule with the Regex value vip and the Nested access pattern
value user returns the following result:
{"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:41.64283645Z","user":{"vip":"no"}}
{"page_id":10,"action":"purchase","timestamp":"2023-03-28T09:08:42.643343109Z","user":{"vip":"yes"}}
{"page_id":50,"action":"click","timestamp":"2023-03-28T09:08:48.643600498Z","user":{"vip":"no"}}
{"page_id":40,"action":"purchase","timestamp":"2023-03-28T09:08:50.643773688Z","user":{"vip":"yes"}}
{"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:51.643932272Z","user":{"vip":"no"}}
{"page_id":40,"action":"click","timestamp":"2023-03-28T09:08:56.644080944Z","user":{"vip":"yes"}}
{"page_id":30,"action":"click","timestamp":"2023-03-28T09:09:03.64425954Z","user":{"vip":"no"}}
{"page_id":20,"action":"view","timestamp":"2023-03-28T09:09:03.644317046Z","user":{"vip":"no"}}
{"page_id":50,"action":"purchase","timestamp":"2023-03-28T09:09:10.64447719Z","user":{"vip":"yes"}}
{"page_id":10,"action":"view","timestamp":"2023-03-28T09:09:17.644810963Z","user":{"vip":"yes"}}
{"page_id":50,"action":"view","timestamp":"2023-03-28T09:09:20.644994805Z","user":{"vip":"no"}}
This rule retained the vip field within the user object and removed all other
fields within user. However, because the processing rule's scope was limited to
user, the rule didn't affect the timestamp, page_id, or action fields.