> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Normalize your log data
During the ingestion phase, after parsing your data, Chronosphere Observability
Platform applies your preferred field normalization rules to populate predefined keys
such as `service`, `severity`, and `timestamp`.
```mermaid actions={false} theme={null}
flowchart LR
Send["Send data"]
subgraph OP["Observability Platform"]
Parse[Parse data]
Normalize[Normalize data]
Control[Control data]
Format[Format data]
Store[Store data]
end
Send --> Parse --> Normalize --> Control --> Format --> Store
click Parse "/control/shaping/shape-logs/parse-logs"
click Control "/control/shaping/shape-logs/control-logs"
click Format "/control/shaping/shape-logs/lookup-logs/"
classDef stages fill:#2fbf71,stroke-width:2px,stroke:#2fbf71,color:#FFFFFF;
class Normalize stages
```
This mapping provides consistency in your data, and optimizes querying in Logs
Explorer. After applying these mappings, Observability Platform applies
[control rules](/control/shaping/shape-logs/control-logs), and then stores your
log data.
In addition to these mappings, you can apply custom field mappings to normalize log
data. For example, you can normalize values from several fields that all relate to different
environments, such as `environment` and `region`. You can then add optional mappings
for individual environments, such as mapping `dev` to `development`, which
standardizes and enriches your log data.
```json theme={null}
"custom_field_normalization": [
{
"target": "environment",
"normalization": {
"source": [
{
"selector": "env"
},
{
"selector": "region"
}
],
"default_value": "production",
"value_map": {
"dev": "development",
"prod": "production",
"stg": "staging"
}
}
}
...
]
```
Custom field mappings are applied after log data is ingested, but before
Observability Platform stores it.
## View custom field mappings
Select from the following methods to view custom field mappings.
To view custom field mappings:
1. In the navigation menu, click ** Go to Admin**
and then select
** Control > Logs Ingest**.
All field mappings display in the **Field normalization** section, including both
predefined and custom field mappings.
2. Choose which custom field mappings to view:
* To view an individual custom field mapping, click it in the list to open the
configuration in the **Edit field mapping** drawer.
* To view the code definition for all configured custom field mappings, click the
**Code config** tab.
To use [Chronoctl](/tooling/chronoctl) to return all custom field mappings, use this
command:
```shell theme={null}
chronoctl log-ingest-config read
```
[Use the Code Config tool](/tooling/gitops#use-the-code-config-tool) in Observability
Platform to view a custom field mappings's Terraform representation.
To complete this action with the Chronosphere API, use the
[`ReadLogIngestConfig`](/tooling/api-info/definition/operations/ReadLogIngestConfig)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
## Create custom field mappings
Use one of the following methods to create custom field mappings.
Define the custom field mapping definition in Observability Platform, and then
[use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply the definition.
1. In the navigation menu, click ** Go to Admin**
and then select
** Control > Logs Ingest**.
2. In the **Field normalization** section, click **Create**.
3. In the **Custom field name** field, enter the custom field name to map data to,
such as `environment`, `region`, or `userID`.
4. In the **Mapped fields** section, in the **Source field name** field, enter a
source field name from your log data to map to the custom field you're creating.
Fields are matched in priority order.
To add more fields, click **Add field mapping** and then enter another source
field.
Use a dot (`.`) to indicate nesting within a field. For example, if your logs are
in a format like `{"resource": {"labels": {"project_id": "..."}}}`, then enter
`resource.labels.project_id`.
If your logs already use dot notation, such as `{"resource.labels.project_id"}`,
enclose the selector in double quotes and brackets like
`["resource.labels.project_id"]`.
5. Optional: In the **Default value** field, enter a default value as the mapping if
the custom field contains no values.
6. Optional: Select either **Regex** or **Value map** as the normalization mode.
* **Regex**: Enter a regular expression pattern to extract and normalize values,
such as for removing extraneous data. For example, use the following regular
expression to remove prefixes and suffixes from any environments including
`production` in the name:
```shell theme={null}
^.*?(production).*?$
```
* **Value map**: Enter key:value pairs to map additional metadata within your
custom field. For example, map `"warn"` to `"Warning"` and `"err"` to `"Error"`
for different severity types.
To add more value mappings, click **Add value mapping** and then define the
value mapping.
7. Click **Save** to save the custom field mapping definition.
8. Click the **Code config** tab and
[use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your
changes.
To use [Chronoctl](/tooling/chronoctl) to create custom field mappings, use this
command:
```shell theme={null}
chronoctl log-ingest-config create
```
1. Run the following command to generate a sample configuration you can use as a
template:
```shell theme={null}
chronoctl log-ingest-config scaffold
```
In the template, `kind: LogIngestConfig` defines an individual log ingest
configuration.
2. Use the `custom_field_normalization` object to define the plain text parser to run
on your log data.
3. With a completed definition, submit it with:
```shell /FILE_NAME/ theme={null}
chronoctl log-ingest-config create -f FILE_NAME
```
Replace *`FILE_NAME`* with the name of the YAML definition file you want to use.
When you run `terraform plan` to generate an execution plan, Chronosphere automatically
tests configurations that include notification policies by submitting them as dry runs.
For details, see the
[Terraform provider](/tooling/infrastructure/terraform#validate-plans-with-dry-runs)
documentation.
To create a custom field mapping with [Terraform](/tooling/infrastructure/terraform):
1. Create a Terraform resource with the `chronosphere_log_ingest_config` resource
type, followed by a resource name.
2. Use the `custom_field_normalization` object to define the custom field mapping to
run on your log data.
3. Add the definition to a Terraform file.
4. Run this command to create the resource:
```shell theme={null}
terraform apply
```
To complete this action with the Chronosphere API, use the
[`CreateLogIngestConfig`](/tooling/api-info/definition/operations/CreateLogIngestConfig)
endpoint.
Use the `"custom_field_normalization"` object to define the custom field mapping to
run on your log data.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
## Edit custom field mappings
When creating or editing a custom field mapping, you can
[use the Code Config tool](/tooling/gitops#use-the-code-config-tool)
to view code representations of a field mapping. The displayed code also responds to
changes you make in the **Visual editor** tab.
Select from the following methods to edit custom field mappings. After selecting a
field mapping to edit, you can change the field mappings order. Fields are matched in
priority order, so an order of `1` has higher precedence over an order of `6`.
1. In the navigation menu, click ** Go to Admin**
and then select
** Control > Logs Ingest**.
2. In the **Field normalization** section, click the custom field mapping you want to
edit.
3. In the **Edit field mapping** drawer, change your custom field mapping.
4. To reorder field mappings, drag and drop them to change the order.
5. Click **Save**. Observability Platform saves changes to your custom field mapping.
6. Click the **Code config** tab and
[use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your
changes.
To update custom field mappings with [Chronoctl](/tooling/chronoctl), use this
command:
```shell theme={null}
chronoctl log-ingest-config update
```
1. Update the custom field mapping definition file.
2. Run the following command to submit the changes:
```shell /FILE_NAME/ theme={null}
chronoctl log-ingest-config update -f FILE_NAME.yaml
```
Replace *`FILE_NAME`* with the name of the YAML definition file you want to use.
Users can modify Terraform-managed resources only by using Terraform.
[Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources).
To edit custom field mappings using [Terraform](/tooling/infrastructure/terraform):
1. Create or edit a Terraform file that updates the resource's existing properties.
2. Run this command to apply the changes:
```shell theme={null}
terraform apply
```
You can also [use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to
view the custom field mapping's Terraform representation.
To complete this action with the Chronosphere API, use the
[`UpdateLogIngestConfig`](/tooling/api-info/definition/operations/UpdateLogIngestConfig)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.
## Delete custom field mappings
Select from one of the following methods to delete custom field mappings.
1. In the navigation menu, click ** Go to Admin**
and then select
** Control > Logs Ingest**.
2. In the **Edit field mapping** drawer, in the row of the field mapping you want to
delete, click the **** trash can icon.
3. Click **Save** to save your changes.
4. Click the **Code config** tab and
[use the Code Config tool](/tooling/gitops#use-the-code-config-tool) to apply your
changes.
Observability Platform deletes your custom field mapping.
To delete a custom field mapping with [Chronoctl](/tooling/chronoctl), use this
command:
```shell theme={null}
chronoctl log-ingest-config delete
```
Users can modify Terraform-managed resources only by using Terraform.
[Learn more](/tooling/infrastructure/terraform#prevent-changes-to-managed-resources).
To delete a resource that's managed by [Terraform](/tooling/infrastructure/terraform):
1. Edit your Terraform configuration file to remove the pre-existing resource
definition.
2. Run this command to remove the resource from Observability Platform:
```shell theme={null}
terraform apply
```
To complete this action with the Chronosphere API, use the
[`DeleteLogIngestConfig`](/tooling/api-info/definition/operations/DeleteLogIngestConfig)
endpoint.
Because the Chronosphere API requires authentication, include an API token with your
`curl` request, as shown in the following example. For more details, see
[Create an API token](/tooling/api-info#create-an-api-token).
```shell /"TOKEN"/ /INSTANCE/ /METHOD/ /ENDPOINT_PATH/ theme={null}
export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"
curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
-X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"
```
Replace the following:
* *`TOKEN`*: Your API token.
* *`INSTANCE`*: The subdomain name for your organization's Observability Platform instance.
* *`METHOD`*: The HTTP method to use with the request, such as `GET` or `POST`.
* *`ENDPOINT_PATH`*: The specific endpoint you want to access.