TELEMETRY PIPELINE
Data
Process
Processing rules
Parse

Parse

The parse processing rule uses a regular expression to search for values inside a string and to assign a key to each value, then stores those key/value pairs in a structured object. This rule mirrors the parsers feature of Chronosphere Telemetry Pipeline.

For a processing rule that performs a similar operation on embedded data already formatted as key/value pairs, see extract keys/values.

Configuration parameters

Use the parameters in this section to configure this processing rule. The Telemetry Pipeline web interface uses the items in the Name column to describe these parameters. Pipeline configuration files use the items in the Key column as YAML keys.

NameKeyDescriptionDefault
Source keysrcRequired. The key whose value contains data to parse.none
Destination keydstRequired. The key of the object to store your structured key/value pairs. This rule can't overwrite an existing key, so this value must be a unique name within your telemetry data.none
RegexregexRequired. The regular expression for extracting values from the value of Source key and assigning keys to those values.none
Regex engineregexEngineRequired. The engine to parse your regular expression. Accepted values: GNU, Oniguruma, PCRE2, POSIX, TRE.PCRE2
CommentcommentA custom note or description of the rule's function. This text is displayed next to the rule's name in the Actions list in the processing rules interface.none

Example

Using the parse processing rule lets you extract any data from a string and turn that data into parsable key/value pairs. You can then use these key/value pairs in other processing rules or for general storage and analysis.

For example, given the following sample website logs:

{"log":"198.143.234.244 - - [18/Mar/2012:13:55:36 -0700] \"GET /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; Trident/4.0)\""}
{"log":"66.216.63.42 - Lucille [18/Mar/2012:15:07:29 -0700] \"DELETE /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_3; like Mac OS X) AppleWebKit/533.11 (KHTML, like Gecko)  Chrome/52.0.3189.332 Mobile Safari/536.3\""}
{"log":"70.137.97.47 - - [18/Mar/2012:19:21:04 -0700] \"GET /lease.pdf HTTP/1.0\" 404 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; Trident/4.0)\""}
{"log":"7.160.29.68 - Michael [19/Mar/2012:01:38:17 -0700] \"POST /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (U; Linux i585 x86_64; en-US) Gecko/20130401 Firefox/60.7\""}
{"log":"31.119.193.169 - - [19/Mar/2012:08:33:29 -0700] \"GET /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (iPad; CPU iPad OS 7_9_0 like Mac OS X) AppleWebKit/602.13 (KHTML, like Gecko)  Chrome/50.0.2327.233 Mobile Safari/534.2\""}

A processing rule with the Source key value log, the Destination key value parsed, the Regex value ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>.*)")?$, and the Regex engine value PCRE2 returns the following result:

{"parsed":{"host":"198.143.234.244","time":"18/Mar/2012:13:55:36 -0700","path":"/lease.pdf","code":"200","user":"-","agent":"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; Trident/4.0)","method":"GET","referer":"bluth-homes.oc","size":"5689972"},"log":"198.143.234.244 - - [18/Mar/2012:13:55:36 -0700] \"GET /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; Trident/4.0)\""}
{"parsed":{"host":"66.216.63.42","time":"18/Mar/2012:15:07:29 -0700","path":"/lease.pdf","code":"200","user":"Lucille","agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_3; like Mac OS X) AppleWebKit/533.11 (KHTML, like Gecko)  Chrome/52.0.3189.332 Mobile Safari/536.3","method":"DELETE","referer":"bluth-homes.oc","size":"5689972"},"log":"66.216.63.42 - Lucille [18/Mar/2012:15:07:29 -0700] \"DELETE /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_3; like Mac OS X) AppleWebKit/533.11 (KHTML, like Gecko)  Chrome/52.0.3189.332 Mobile Safari/536.3\""}
{"parsed":{"host":"70.137.97.47","time":"18/Mar/2012:19:21:04 -0700","path":"/lease.pdf","code":"404","user":"-","agent":"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; Trident/4.0)","method":"GET","referer":"bluth-homes.oc","size":"5689972"},"log":"70.137.97.47 - - [18/Mar/2012:19:21:04 -0700] \"GET /lease.pdf HTTP/1.0\" 404 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.3; Trident/4.0)\""}
{"parsed":{"host":"7.160.29.68","time":"19/Mar/2012:01:38:17 -0700","path":"/lease.pdf","code":"200","user":"Michael","agent":"Mozilla/5.0 (U; Linux i585 x86_64; en-US) Gecko/20130401 Firefox/60.7","method":"POST","referer":"bluth-homes.oc","size":"5689972"},"log":"7.160.29.68 - Michael [19/Mar/2012:01:38:17 -0700] \"POST /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (U; Linux i585 x86_64; en-US) Gecko/20130401 Firefox/60.7\""}
{"parsed":{"host":"31.119.193.169","time":"19/Mar/2012:08:33:29 -0700","path":"/lease.pdf","code":"200","user":"-","agent":"Mozilla/5.0 (iPad; CPU iPad OS 7_9_0 like Mac OS X) AppleWebKit/602.13 (KHTML, like Gecko)  Chrome/50.0.2327.233 Mobile Safari/534.2","method":"GET","referer":"bluth-homes.oc","size":"5689972"},"log":"31.119.193.169 - - [19/Mar/2012:08:33:29 -0700] \"GET /lease.pdf HTTP/1.0\" 200 5689972 \"bluth-homes.oc\" \"Mozilla/5.0 (iPad; CPU iPad OS 7_9_0 like Mac OS X) AppleWebKit/602.13 (KHTML, like Gecko)  Chrome/50.0.2327.233 Mobile Safari/534.2\""}

This rule extracted values from the string stored in the log key, assigned a key to each value, then stored the resulting key/value pairs in a new structured object named parsed.

Nest keysParse number