Custom Lua

You can use the custom Lua processing rule to write your own Lua scripts for transforming telemetry data.

This rule uses LuaJIT 5.1 (opens in a new tab) and runs in protected mode to ensure that malformed Lua code doesn't harm pipelines.

Configuration parameters

Use the parameters in this section to configure this processing rule. The Telemetry Pipeline web interface uses the items in the Name column to describe these parameters. Pipeline configuration files use the items in the Key column as YAML keys.

GPT generation

The custom Lua rule processing rule lets you turn text descriptions into scripts by using generative pre-trained transformer (GPT) generation. To use this feature, enter a description of your desired script into the Comment text box, and then click Generate Lua script from comment.

Best practices

To ensure that custom Lua scripts run efficiently within your pipelines, Chronosphere recommends following these best practices:

• Test new scripts in the processing rules playground. Before adding a custom Lua script to an active pipeline, try running the script in the processing rules playground in the Telemetry Pipeline UI. This lets you test your script on sample input data, view the script's transformed output, and catch syntax errors.

• Add print and error statements to your scripts. These statements help you track the value of variables and see details about specific errors when they occur.

• Use profiling. In the processing rules playground, you can toggle the Enable profiling setting to view more information about a script's transformed output, including how long it took for that script to run.

Example scripts

The following examples show several possible types of Lua scripts you can incorporate into your pipeline.

Conditional statement

You can use conditional statements to transform data that meets certain criteria. This example script looks for a key named log, and then determines whether its value is line. If both of these conditions are true, the script sets the value of log to Y.

return function(tag, ts, record)
    if record['log'] ~= nil and record['log'] == 'line' then
        record['log'] = 'Y'
    end
    return 1, ts, record
end

Reference to a secret

If your pipeline includes secrets, you can reference these secrets through Lua scripts. Because secrets are stored as environment variables within a pipeline, this example accesses that environment variable, and then stores its value in a new variable named secret.

Additionally, this script calls the resolve function only if the secret key hasn't been defined before, which helps improve performance.

function resolve(SECRET_<STR>)
  local token = os.getenv(SECRET_<STR>)
  if token == nill
    then error(string.format("Could not resolve token"))
  else
    return token
  end
end
 
local secret = nil
 
return function(tag, ts, record, code)
  if secret == nil then
    secret = resolve("SECRET_<STR>")
    record.new_field = secret
  else
    record.new_field = secret
  end
  return code, ts, record
end

Call to a third-party API

You can use Lua scripts to call a third-party HTTP API, and then use data from that API to perform various actions.

This example imports the Lua JSON library to support the JSON response returned by a specific API, then uses a separate function to call that API, and then creates a new key to store the value of a successful API response.

local json = require 'json'
 
local function try_curl(url, max_retries, timeout)
  local retries = 0
  while retries < max_retries do
    local success, result = pcall(function()
      local command = string.format('curl -s -L --max-time %d %s', timeout, url)
      local proc = io.popen(command)
      local output = proc:read('*a')
      proc:close()
      return output
    end)
 
    if success and result ~= "" then
      return result
    end
 
    retries = retries + 1
    if retries < max_retries then
      os.execute("sleep 1")  -- Wait for 1 second before retrying
    end
  end
  return nil
end
 
return function(tag, ts, record, code)
  if record.___dummyFlush__ then
    return 1, ts, record
  end
 
 
  local url = 'https://gist.githubusercontent.com/agup006/7fbc70a9a821bb39977504afb9ec290e/raw/9ced93796b9eb5a8d0b0d69bdeed630d5c7f2744/test.json'
  local max_retries = 3
  local timeout = 10  -- seconds
 
  local response = try_curl(url, max_retries, timeout)
 
  if response then
    local success, result = pcall(json.decode, response)
    if success then
      record.json = result
    else
      record.json = { message = "failed to parse JSON" }
    end
  else
    record.json = { message = "failed to connect" }
  end
 
  return code, ts, record
end

Reading a pipeline file

You can use pipeline files to store arbitrary data, including JSON, and use a Lua script to access that data.

This example imports the Lua JSON library, then uploads a file called schema, and then converts and returns the data stored in that file.

local json = require 'json'
 
local file_path = "/config/schema"
local file = io.open(file_path, "r")
if not file then
    error("Could not open file at " .. file_path)
end
 
local jsonschema_content = file:read("*all")
file:close()
 
local jsonschema = json.decode(jsonschema_content)
if not jsonschema then
    error("Failed to parse JSON from file at " .. file_path)
end
 
return function(tag, ts, record, code)
  if record.___dummyFlush__ then
    return 1, ts, record
  end
 
  return code, ts, jsonschema
end