Custom pipelines
Processing rules
Allow keys

Allow keys action

The allow keys action is used in processing rules to preserve any keys that match a specified regular expression, and remove all other keys.

For an action with the opposite effect, see block keys.

Configuration parameters

  • Regex: Required. The regular expression that determines which keys to preserve.
  • Match case: Indicates if the regular expression is case-sensitive.
  • Regex engine: Required. The engine to parse your regular expression. Default: PCRE2.
  • Nested access pattern: The key of the highest object level that you'd like to rule to evaluate. If specified, the rule ignores any keys not contained within the specified object. If unspecified, the allow keys rule evaluates all top-level keys, but no keys nested within top-level keys. You can also use record accessor syntax to reference keys nested within another nested object.
  • Comment: A description for the action's function that is displayed in the Actions list.

Examples

Using the allow keys action lets you pare down telemetry data by retaining only the data you explicitly choose to keep.

Flat example

For example, given this sample log data:

{"timestamp":"2023-03-28T09:08:41.64283645Z","user_id":3,"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:42.643343109Z","user_id":4,"page_id":10,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:48.643600498Z","user_id":1,"page_id":50,"action":"click"}
{"timestamp":"2023-03-28T09:08:50.643773688Z","user_id":5,"page_id":40,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:51.643932272Z","user_id":1,"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:56.644080944Z","user_id":2,"page_id":40,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.64425954Z","user_id":3,"page_id":30,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.644317046Z","user_id":1,"page_id":20,"action":"view"}
{"timestamp":"2023-03-28T09:09:10.64447719Z","user_id":2,"page_id":50,"action":"purchase"}
{"timestamp":"2023-03-28T09:09:17.644810963Z","user_id":2,"page_id":10,"action":"view"}
{"timestamp":"2023-03-28T09:09:20.644994805Z","user_id":1,"page_id":50,"action":"view"}

A processing rule with the Regex value user returns the following result:

{"user_id":3}
{"user_id":4}
{"user_id":1}
{"user_id":5}
{"user_id":1}
{"user_id":2}
{"user_id":3}
{"user_id":1}
{"user_id":2}
{"user_id":2}
{"user_id":1}

Nested example

You can also use the allow keys action to selectively retain information within a nested object. For example, given this sample log data:

{"timestamp":"2023-03-28T09:08:41.64283645Z","user":{"vip":"no","id":3},"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:42.643343109Z","user":{"vip":"yes","id":4},"page_id":10,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:48.643600498Z","user":{"vip":"no","id":1},"page_id":50,"action":"click"}
{"timestamp":"2023-03-28T09:08:50.643773688Z","user":{"vip":"yes","id":5},"page_id":40,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:51.643932272Z","user":{"vip":"no","id":1},"page_id":30,"action":"purchase"}
{"timestamp":"2023-03-28T09:08:56.644080944Z","user":{"vip":"yes","id":2},"page_id":40,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.64425954Z","user":{"vip":"no","id":3},"page_id":30,"action":"click"}
{"timestamp":"2023-03-28T09:09:03.644317046Z","user":{"vip":"no","id":1},"page_id":20,"action":"view"}
{"timestamp":"2023-03-28T09:09:10.64447719Z","user":{"vip":"yes","id":2},"page_id":50,"action":"purchase"}
{"timestamp":"2023-03-28T09:09:17.644810963Z","user":{"vip":"yes","id":2},"page_id":10,"action":"view"}
{"timestamp":"2023-03-28T09:09:20.644994805Z","user":{"vip":"no","id":1},"page_id":50,"action":"view"}

A processing rule with the Regex value vip and the Nested access pattern value user returns the following result:

{"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:41.64283645Z","user":{"vip":"no"}}
{"page_id":10,"action":"purchase","timestamp":"2023-03-28T09:08:42.643343109Z","user":{"vip":"yes"}}
{"page_id":50,"action":"click","timestamp":"2023-03-28T09:08:48.643600498Z","user":{"vip":"no"}}
{"page_id":40,"action":"purchase","timestamp":"2023-03-28T09:08:50.643773688Z","user":{"vip":"yes"}}
{"page_id":30,"action":"purchase","timestamp":"2023-03-28T09:08:51.643932272Z","user":{"vip":"no"}}
{"page_id":40,"action":"click","timestamp":"2023-03-28T09:08:56.644080944Z","user":{"vip":"yes"}}
{"page_id":30,"action":"click","timestamp":"2023-03-28T09:09:03.64425954Z","user":{"vip":"no"}}
{"page_id":20,"action":"view","timestamp":"2023-03-28T09:09:03.644317046Z","user":{"vip":"no"}}
{"page_id":50,"action":"purchase","timestamp":"2023-03-28T09:09:10.64447719Z","user":{"vip":"yes"}}
{"page_id":10,"action":"view","timestamp":"2023-03-28T09:09:17.644810963Z","user":{"vip":"yes"}}
{"page_id":50,"action":"view","timestamp":"2023-03-28T09:09:20.644994805Z","user":{"vip":"no"}}

This rule retained the vip field within the user object and removed all other fields within user. However, because the processing rule's scope was limited to user, the rule didn't affect the timestamp, page_id, or action fields.

Add/set key/valueAllow records