Datagen source plugin - Chronosphere Documentation

This information is for Chronosphere Telemetry Pipeline, which is a standalone product separate from Chronosphere Observability Platform.

The Datagen source plugin (name: datagen, alias: apache_common) generates simulated log data through Gofakeit. You can use this simulated data to test your pipeline configurations. Although this is a self-contained plugin that doesn’t communicate with external sources, it’s still classified as a pull-based source plugin.

This plugin doesn’t support the use of a descriptive metadata name in the Pipeline Builder interface.

This plugin doesn’t support duplicates of itself within the same pipeline.

Supported telemetry types

The for Chronosphere Telemetry Pipeline supports these telemetry types:

Logs	Metrics	Traces

Configuration parameters

Use the parameters in this section to configure the . The Telemetry Pipeline web interface uses the items in the Name column to describe these parameters. Pipeline configuration files use the items in the Key column as YAML keys.

General

Name	Key	Description	Default
Template	`template`	Required. A JSON object that specifies how to generate log data. Accepted types: Apache common, Syslog rfc3164, or Syslog rfc5424. See Templates.	none
none	`rate`	How often to generate logs, in seconds.	`1`

Advanced

Name	Key	Description	Default
Memory Buffer Limit	`mem_buf_limit`	Sets a limit for how much buffered data the plugin can write to memory, which affects backpressure. This value must follow Fluent Bit’s rules for unit sizes. If unspecified, no limit is enforced. For v2 pipelines, this parameter affects only pipelines with the Deployment or DaemonSet workload type. To learn more, see the v2 backpressure guide. For v3 pipelines, this parameter is independent from the OpenTelemetry `memory_limiter` and `batch` processors.	none

Templates

Templates define the structure and content of your simulated log data. When you configure the Datagen source plugin in a pipeline’s YAML configuration file, use the template key to create a JSON object. For example:

pipeline:
  inputs:
    - name: datagen
      template: |-
        {
            "host":            "{{ IPv4Address }}",
            "user": {
              "user-identifier": "{{ Username }}",
              "auth-user-id":    "{{ UUID }}",
            }
            "method":          "{{ HTTPMethod }}",
            "request":         "{{ URLPath }}",
            "protocol":        "{{ HTTPVersion }}",
            "response-code":   "{{ HTTPStatusCode }}",
            "bytes":           "{{ Number 0 30_000 }}"
        }

When the Datagen plugin generates logs, each log matches the format of your template, but any included functions are replaced by simulated data. For best results, Chronosphere recommends creating a JSON object that uses custom strings for its keys and functions for its values.

Functions

The Datagen plugin uses functions to generate randomized values that follow a consistent pattern. For example, {{ HTTPMethod }} can output a string like GET, POST, or PUT, but not a string like GREEN or SUNDAY. To invoke a function in a template, surround the name of that function with double curly braces.

Suggested functions

You can use any of the available Gofakeit functions in a template. The following functions are best suited for generating realistic log data:

{{ HTTPMethod }}: Returns an HTTP method, like GET.
{{ HTTPStatusCode }}: Returns an HTTP status code, like 403.
{{ HTTPVersion }}: Returns an HTTP protocol version, like HTTP/1.1.
{{ IPv4Address }}: Returns a random IPv4 address.
{{ IPv6Address }}: Returns a random IPv6 address.
{{ Username }}: Returns a random username, like Bailey1270.
{{ UUID }}: Returns a string of numbers in UUID format.
{{ Date }}: Returns a random time in UTC format.
{{ LogLevel syslog }}: Returns a log type, like alert.
{{ Number 0 30_000 }}: Returns a number between 0 and 30,000.
{{ URLPath }}: Returns a string that resembles a URL path, like /foo/bar. This function is unique to Datagen and isn’t included in Gofakeit.
{{ URLScheme }}: Returns either http or https. This function is unique to Datagen and isn’t included in Gofakeit.

Example output

For a pipeline with the following template:

pipeline:
  inputs:
    - name: datagen
      template: |-
        {
            "response-code":   "{{ HTTPStatusCode }}",
            "bytes":           "{{ Number 0 30_000 }}"
            "host":            "{{ IPv4Address }}",
        }

The Datagen source plugin generates simulated log data in the following format:

{"response-code"=>201.000000, "bytes"=>29600.000000, "host"=>"64.15.54.123"}

{"response-code"=>302.000000, "bytes"=>429.000000, "host"=>"214.221.240.201"}

{"response-code"=>205.000000, "bytes"=>3507.000000, "host"=>"27.79.197.250"}

{"response-code"=>504.000000, "bytes"=>758.000000, "host"=>"211.135.114.51"}

{"response-code"=>404.000000, "bytes"=>12190.000000, "host"=>"84.74.177.0"}

​Supported telemetry types

​Configuration parameters

​General

​Advanced

​Templates

​Functions

​Suggested functions

​Example output

Supported telemetry types

Configuration parameters

General

Advanced

Templates

Functions

Suggested functions

Example output