Normalize your log data

During the ingestion phase, after parsing your data, Chronosphere Observability Platform applies your preferred field normalization rules to populate predefined keys such as service, severity, and timestamp. This mapping provides consistency in your data, and optimizes querying in Logs Explorer. After applying these mappings, Observability Platform applies control rules, and then stores your log data. In addition to these mappings, you can apply custom field mappings to normalize log data, such as normalizing values from several fields that all relate to different environments. See Examples for examples of how you can use custom field mappings. Custom field mappings are applied after log data is ingested, but before Observability Platform stores it. To indicate nesting within a source field, use a dot (.). For example, if your logs are in a format like {"resource": {"labels": {"project_id": "..."}}}, then enter resource.labels.project_id. If your logs already use dot notation, such as {"resource.labels.project_id": "..."}, enclose the selector in double quotes and brackets like ["resource.labels.project_id"]. See target nested or flat fields with dot notation for additional examples.

View custom field mappings

Select from the following methods to view custom field mappings.

Web
Chronoctl
Terraform
API

To view custom field mappings:

In the navigation menu, click Go to Admin and then select Control > Logs Ingest. All field mappings display in the Field normalization section, including both predefined and custom field mappings.
Choose which custom field mappings to view:
- To view an individual custom field mapping, click it in the list to open the configuration in the Edit field mapping drawer.
- To view the code definition for all configured custom field mappings, click the Code config tab.

To use Chronoctl to return all custom field mappings, use this command:

chronoctl log-ingest-config read

To complete this action with the Chronosphere API, use the ReadLogIngestConfig endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Create custom field mappings

Use one of the following methods to create custom field mappings.

Web
Chronoctl
Terraform
API

Define the custom field mapping definition in Observability Platform, and then use the Code Config tool to apply the definition.

In the navigation menu, click Go to Admin and then select Control > Logs Ingest.
In the Field normalization section, click Create.
In the Custom field name field, enter the custom field name to map data to, such as environment, region, or userID.
In the Mapped fields section, in the Source field name field, enter a source field name from your log data to map to the custom field you’re creating. Fields are matched in priority order. To add more fields, click Add field mapping and then enter another source field.
Use a dot (.) to indicate nesting within a field. For example, if your logs are in a format like {"resource": {"labels": {"project_id": "..."}}}, then enter resource.labels.project_id.If your logs already use dot notation, such as {"resource.labels.project_id"}, enclose the selector in double quotes and brackets like ["resource.labels.project_id"].
Optional: In the Default value field, enter a default value as the mapping if the custom field contains no values.
Optional: Select either Regex or Value map as the normalization mode.
- Regex: Enter a regular expression pattern to extract and normalize values, such as for removing extraneous data. For example, use the following regular expression to remove prefixes and suffixes from any environments including production in the name:
  ^.*?(production).*?$
- Value map: Enter key:value pairs to map additional metadata within your custom field. For example, map "warn" to "Warning" and "err" to "Error" for different severity types. To add more value mappings, click Add value mapping and then define the value mapping.
Click Save to save the custom field mapping definition.
Click the Code config tab and use the Code Config tool to apply your changes.

To use Chronoctl to create custom field mappings, use this command:

chronoctl log-ingest-config create

Run the following command to generate a sample configuration you can use as a template:
chronoctl log-ingest-config scaffold
In the template, kind: LogIngestConfig defines an individual log ingest configuration.
Use the custom_field_normalization object to define the plain text parser to run on your log data.
With a completed definition, submit it with:
chronoctl log-ingest-config create -f FILE_NAME
Replace FILE_NAME with the name of the YAML definition file you want to use.

When you run terraform plan to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the Terraform provider documentation.

To create a custom field mapping with Terraform:

Create a Terraform resource with the chronosphere_log_ingest_config resource type, followed by a resource name.
Use the custom_field_normalization object to define the custom field mapping to run on your log data.
Add the definition to a Terraform file.
Run this command to create the resource:
terraform apply

To complete this action with the Chronosphere API, use the CreateLogIngestConfig endpoint.Use the "custom_field_normalization" object to define the custom field mapping to run on your log data.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Edit custom field mappings

When creating or editing a custom field mapping, you can use the Code Config tool to view code representations of a field mapping. The displayed code also responds to changes you make in the Visual editor tab. Select from the following methods to edit custom field mappings. After selecting a field mapping to edit, you can change the field mappings order. Fields are matched in priority order, so an order of 1 has higher precedence over an order of 6.

Web
Chronoctl
Terraform
API

In the navigation menu, click Go to Admin and then select Control > Logs Ingest.
In the Field normalization section, click the custom field mapping you want to edit.
In the Edit field mapping drawer, change your custom field mapping.
To reorder field mappings, drag and drop them to change the order.
Click Save. Observability Platform saves changes to your custom field mapping.
Click the Code config tab and use the Code Config tool to apply your changes.

To update custom field mappings with Chronoctl, use this command:

chronoctl log-ingest-config update

Update the custom field mapping definition file.
Run the following command to submit the changes:
chronoctl log-ingest-config update -f FILE_NAME.yaml
Replace FILE_NAME with the name of the YAML definition file you want to use.

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To edit custom field mappings using Terraform:

Create or edit a Terraform file that updates the resource’s existing properties.
Run this command to apply the changes:
terraform apply

You can also use the Code Config tool to view the custom field mapping’s Terraform representation.

To complete this action with the Chronosphere API, use the UpdateLogIngestConfig endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Delete custom field mappings

Select from one of the following methods to delete custom field mappings.

Web
Chronoctl
Terraform
API

In the navigation menu, click Go to Admin and then select Control > Logs Ingest.
In the Edit field mapping drawer, in the row of the field mapping you want to delete, click the trash can icon.
Click Save to save your changes.
Click the Code config tab and use the Code Config tool to apply your changes.

Observability Platform deletes your custom field mapping.

To delete a custom field mapping with Chronoctl, use this command:

chronoctl log-ingest-config delete

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To delete a resource that’s managed by Terraform:

Edit your Terraform configuration file to remove the pre-existing resource definition.
Run this command to remove the resource from Observability Platform:
terraform apply

To complete this action with the Chronosphere API, use the DeleteLogIngestConfig endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Target nested or flat fields with dot notation

The source selectors for custom field mappings support dot notation to read values from fields inside nested structures. How Observability Platform interprets a selector depends on whether you enclose the segments in brackets in double quotes:

An unbracketed dot splits the selector into path segments, which traverses nested fields. A source of metadata.code reads a code field nested inside a metadata object.
A bracket-quoted segment such as ["metadata.code"] treats the quoted string as a single, literal field name that contains a dot. This reads a flat field named metadata.code instead of a nested structure.

Consider a custom field mapping that populates a status_code target from a source field. To read from a nested field, set the source selector to metadata.code:

{
  "target": "status_code",
  "normalization": {
    "source": [
      { "selector": "metadata.code" }
    ]
  }
}

This selector reads the value from the nested code field:

{
  "metadata": {
    "code": "500"
  }
}

To read from a single flat field whose name contains a dot, enclose the source selector in double quotes and brackets:

{
  "target": "status_code",
  "normalization": {
    "source": [
      { "selector": "[\"metadata.code\"]" }
    ]
  }
}

This selector reads the value from one literal field name:

{
  "metadata.code": "500"
}

Examples

The following examples show common custom_field_normalization patterns. Each object is an element of the custom_field_normalization array.

Standardize and enrich data

Normalize values from several fields that all relate to different environments, such as environment and region. You can then add optional mappings for individual environments, such as mapping dev to development.

{
  "target": "environment",
  "normalization": {
    "source": [
      {
        "selector": "env"
      },
      {
        "selector": "region"
      }
    ],
    "default_value": "production",
    "value_map": {
      "dev": "development",
      "prod": "production",
      "stg": "staging"
    }
  }
}

Consolidate values from multiple fields

Map a single region field from whichever source field is present, checking region first and then datacenter. If neither contains a value, apply a default value.

{
  "target": "region",
  "normalization": {
    "source": [
      { "selector": "region" },
      { "selector": "datacenter" }
    ],
    "default_value": "us-east-1"
  }
}

Standardize values with a value map

Normalize inconsistent HTTP status codes into a smaller set of status classes, and label anything unmatched as unknown.

{
  "target": "status_class",
  "normalization": {
    "source": [
      { "selector": "http.status_code" }
    ],
    "value_map": {
      "200": "success",
      "201": "success",
      "400": "client_error",
      "404": "client_error",
      "500": "server_error",
      "503": "server_error"
    },
    "default_value": "unknown"
  }
}

Extract a value with a regular expression

Use sanitize_patterns to extract a substring from a larger field. Each pattern must contain exactly one capturing group, and Observability Platform uses that group as the result. This example extracts an uppercase log level from a free-form message field.

{
  "target": "log_level",
  "normalization": {
    "source": [
      { "selector": "message" }
    ],
    "sanitize_patterns": ["^.*level=([A-Z]+).*$"],
    "default_value": "INFO"
  }
}

Map a value from a nested field

Use dot notation to read a value from a nested structure, such as user.id inside {"user": {"id": "..."}}. Sources are still matched in priority order.

{
  "target": "userID",
  "normalization": {
    "source": [
      { "selector": "user.id" },
      { "selector": "actor.userId" }
    ]
  }
}

Map a value from a flat field that contains dots

If a source field name already contains dots, bracket-quote the selector so Observability Platform treats it as one literal field name instead of a nested path.

{
  "target": "team",
  "normalization": {
    "source": [
      { "selector": "[\"kubernetes.labels.team\"]" }
    ],
    "default_value": "unassigned"
  }
}

Observability Platform

Telemetry Pipeline

More information

View custom field mappings

Create custom field mappings

Edit custom field mappings

Delete custom field mappings

Target nested or flat fields with dot notation

Examples

Standardize and enrich data

Consolidate values from multiple fields

Standardize values with a value map

Extract a value with a regular expression

Map a value from a nested field

Map a value from a flat field that contains dots

​View custom field mappings

​Create custom field mappings

​Edit custom field mappings

​Delete custom field mappings

​Target nested or flat fields with dot notation

​Examples

​Standardize and enrich data

​Consolidate values from multiple fields

​Standardize values with a value map

​Extract a value with a regular expression

​Map a value from a nested field

​Map a value from a flat field that contains dots

View custom field mappings

Create custom field mappings

Edit custom field mappings

Delete custom field mappings

Target nested or flat fields with dot notation

Examples

Standardize and enrich data

Consolidate values from multiple fields

Standardize values with a value map

Extract a value with a regular expression

Map a value from a nested field

Map a value from a flat field that contains dots