Control your log data

Chronosphere Observability Platform provides control rules that let you manage log data by transforming, reshaping, retaining, or excluding data before it’s stored. These rules let you control the volume of persisted log data and improve data quality. By controlling which data you keep or drop, you can reduce noise, better control costs, and focus on data that matters most to your business.

Applying control rules

Control rules run after both parsers and field mappings run. Disabled rules are skipped. Enable rules to have Observability Platform evaluate them. Observability Platform applies control rules serially and in order. Each rule operates on the output of the rule before it. Effects of previous rules apply immediately to the next rule in the sequence. If a rule matches the same condition as a previous rule, the subsequent rule operates only on the outcome of the previous rule. For example, if the first rule samples logs by fifty percent where service = "auth" and the second rule also includes this condition, the second rule evaluates only the remaining fifty percent of the original data volume. Observability Platform includes the following control rules:

Drop logs that match a specific filter to remove unnecessary or low-value data.
Drop fields from logs that match specified conditions, which helps reduce the size of individual logs. If a log contains no data after fields are dropped, you can choose to drop the entire log.
Sample logs to retain a certain percentage of matching logs and drop the remainder, which can significantly reduce volume and maintain visibility.
Emit metric converts log data to metric data. Use this control rule when you want to reduce the volume of logs, retain data for a longer period, and increase the speed of querying and filtering for metrics instead of logs.
Replace field identifies a specified field in your log data and uses a regular expression to replace any matches based on the selected replacement mode. Use this control rule to truncate long fields and preserve your log structure.

To change the order that control rules are applied, reorder rules.

View control rules

Select from one of the following methods to view control rules.

Web
Chronoctl
Terraform
API

To view control rules:

In the navigation menu, click Go to Admin and then select Control > Logs Ingest. All defined control rules display in the Control rules section.
To view only control rules, from the View dropdown, select Control rules.
To view the code definition for all configured control rules, click the Code config tab. The Logs control rules section includes all defined control rules.

To use Chronoctl to return all control rules, use the chronoctl log-control-config read command:

chronoctl log-control-config read

To complete this action with the Chronosphere API, use the ReadLogControlConfig endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Create control rules

Use control rules to drop certain logs at ingestion, or fields that you don’t want to persist. For example, dropping log data from a specific cluster, test data in a development environment, or unnecessary individual fields across your logs. When creating control rules, be sure you understand how Observability Platform applies control rules. Use one of the following methods to create control rules for log data:

Web
Chronoctl
Terraform
API

Define control rules in Observability Platform, and then use the Code Config tool to apply the rule definition.

In the navigation menu, click Go to Admin and then select Control > Logs Ingest.
In the Control rules section, click Create.
In the Create control rule page, enter a name for the control rule.
From the Action menu, select the type of rule to create:
- Drop logs: Completely drop logs that match the specified filter.
- Drop fields: Remove specific fields from logs that match the specified filter.
- Sample logs: Retain a certain percentage of matching logs and drop the remainder.
- Emit metric: Convert log data to metric data.
- Replace field: Replace context within a log field using a regular expression.
Enter a filter to return log data for the control rule, and then press Ctrl+Enter (Command+Return on macOS) to submit the filter. This rule applies only to logs that match this filter at the time the log data was ingested.
Review the returned data from the preview filter and make changes as necessary.
Control rules are applied during ingestion, so the preview filter results might differ from a filter submitted in Logs Explorer after ingestion. By default, including a dot (.) in a filter indicates nesting within a field.In some instances, fields with a key that uses dot notation within the name, such as resource.type, must be surrounded by double quotes and brackets in the control rule filter.If a warning message displays indicating that This filter is applied at ingestion and the filter returns no results, wrap the key in double quotes ( ") and brackets ([]). For example, the following filter contains the resource.type key, which requires surrounding the field in double quotes and brackets for this particular service:
service = "nginx" and severity = "INFO" and ["resource.type"] = "k8s_container"
Enter additional information for the specified action:
- Drop logs
- Drop fields
- Sample logs
- Emit metric
- Replace fields
Review the filter and make changes as needed.
Define the criteria for the drop rule:
- In the Parent path field, enter the level of depth in the log to apply the regular expression to. For example, if you wanted to apply the regular expression to fields under httpRequest, such as httpRequest.status, enter httpRequest in the Parent path field. For fields with no nesting, leave the Parent path field empty.
- In the Field regex field, enter the regular expression that determines which fields to match on.
In the Logs to keep field, enter a percentage of logs you want to keep. For example, 25 for 25%.
Define the criteria for the rule to emit metrics from logs:
- Enter a unique name for the generated metric.
- Select the metric mode for the generated metric:
  
  Counter: A metric that counts the number of times a certain event has occurred.
  
  Gauge: A metric that tracks a value which changes over time. Choose Last (most recent), Min (minimum value), or Max (maximum value) to define which value to use.
  
  Exponential Histogram: A metric that tracks the distribution of values across data groupings with exponentially increasing size.
- Enter a name for the log field whose value you want to use as the metric value. Use field[field] syntax to indicate nested fields.
- Enter labels as key/value pairs to add to the metric.
- To drop the original log, select the Drop original log after emitting metric checkbox.
Define the criteria for the replace rule:
- Enter the name of the log field to apply the rule to. In the Match regex field, enter the regular expression that determines which data to match on in the specified field.
- Determine whether to replace the first match only (Replace first), or to replace all matches (Replace all).
- Select the replacement mode:
  
  Static value: Replace the matched content with a static string. For example, replace punctuation in a field with an empty string, or truncate the ends of long stack traces.
  
  Mapped value: Replace the matched content with specified key/value pairs. For example, reduce log volume by replacing a string error with a specific error code. If none of the key/value pairs match, the provided default value is used. Click Provide default value to specify a default value if no matching key is found.
  
  Hash: Replace the matched content with a hashed string, which can help reduce the size of large strings. After replacing the original content with a string, there’s no way to recover that information.
Click Save to save the control rule definition.
Click the Code config tab and use the Code Config tool to apply the definition.

To use Chronoctl to create control rules, use the chronoctl log-control-config create command:

chronoctl log-control-config create

Run the following command to generate a sample control rule configuration you can use as a template:
```
chronoctl log-control-config scaffold
```
In the template, kind: LogControlConfig defines an individual control rule configuration.
With a completed definition, submit it with:
```
chronoctl log-control-config create -f FILE_NAME
```
Replace FILE_NAME with the name of the YAML definition file you want to use.

See the Chronoctl example for more information.

When you run terraform plan to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the Terraform provider documentation.

To create a control rule with Terraform:

Create a Terraform resource with the chronosphere_log_control_config resource type, followed by a resource name.
Add the definition to a Terraform file.
Run this command to create the resource:
```
terraform apply
```

See the Terraform example for more information.

To complete this action with the Chronosphere API, use the CreateLogControlConfig endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Chronoctl example

The following example defines a control rules file for Chronoctl, and contains the following rules:

The first rule samples 10% of logs that match the defined filter.
The second rule drops all logs for the nginx service of severity INFO where the http.Request.status equals 200.
The third rule drops all kubernetes fields that match the defined filter. The regular expression indicates that any fields matching kubernetes.[FIELD] are dropped from all logs.
The fourth rule uses a mapped value to search for the error_code field, and replaces values based on the provided key/value pairs. If the rule matches on INTERNAL_SERVER_ERROR, it replaces that value with 500. If the rule matches on NOT_FOUND, it replaces the value with 401.
The fifth rule emits a counter metric from NGINX error logs. Each matching log increments nginx_errors_total by one. Labels on the metric come from the service and error_code log fields.

api_version: v1/config
kind: LogControlConfig
spec:
  # The ordered list of control rules.
  rules:
    # First rule
    - name: "Sample billing service with missing user ID"
      mode: ENABLED
      # Log filter syntax to select logs. Only matching logs will have control action applied.
      filter: "cluster_name:'production'\n
              AND service='billing-service'\n
              AND message = 'missing user ID'"
      type: SAMPLE
      sample:
        # Percentage of matching logs to keep. Must be in the range [0, 1].
        rate: 0.10
    # Second rule
    - name: "Drop NGINX information logs"
      mode: ENABLED
      filter: "service = 'nginx' and severity = 'INFO' and httpRequest.status = '200'"
      type: DROP
    # Third rule
    - name: 'Drop Kubernetes fields from development environments'
      mode: ENABLED
      filter: "namespace = 'development' and kubernetes.cluster_name = 'test'"
      # The configuration for a drop field action.
      type: DROP_FIELD
      drop_field:
        # Regular expression to match the field names to drop.
        field_regex: .*
        # Selector to indicate field path. Use 'parent[child]' syntax to
        # indicate nesting.
        parent_path:
          selector: kubernetes
    # Fourth rule
    - name: error replacement
      mode: ENABLED
      filter: service = "nginx" and severity = "ERROR"
      type: REPLACE_FIELD
      replace_field:
        field:
          selector: error_code
        replace_regex: .*
        replace_all: true
        replace_mode: MAPPED_VALUE
        mapped_value:
          pairs:
            - key: INTERNAL_SERVER_ERROR
              value: "500"
            - key: NOT_FOUND
              value: "401"
    # Fifth rule
    - name: "Emit counter for NGINX errors"
      mode: ENABLED
      filter: service = "nginx" and severity = "ERROR"
      type: EMIT_METRICS
      emit_metrics:
        # Prometheus-compatible metric name.
        name: nginx_errors_total
        mode: COUNTER
        labels:
          - key: service
            value:
              selector: service
          - key: error_code
            value:
              selector: error_code

Terraform example

The following example defines a control rules resource for Terraform, and contains the following rules:

The first rule samples 10% of logs that match the defined filter.
The second rule drops all logs for the nginx service of severity INFO where the http.Request.status equals 200.
The third rule drops all kubernetes fields that match the defined filter. The regular expression indicates that any fields matching kubernetes.[FIELD] are dropped from all logs.
The fourth rule uses a mapped value to search for the error_code field, and replaces values based on the provided key/value pairs. If the rule matches on INTERNAL_SERVER_ERROR, it replaces that value with 500. If the rule matches on NOT_FOUND, it replaces the value with 401.
The fifth rule emits a counter metric from NGINX error logs. Each matching log increments nginx_errors_total by one. Labels on the metric come from the service and error_code log fields.

resource "chronosphere_log_control_config" "log_control_config" {
  # The ordered list of control rules.
  rules {
    # First rule
    name   = "Sample billing service with missing user ID"
    # Log filter syntax to select logs. Only matching logs will have control action applied.
    filter = chomp(<<-EOF
    cluster_name:"rc"
    AND service="billing-service"
    AND message = "missing user ID"
    EOF
    )
    mode = "ENABLED"

    sample {
      # Percentage of matching logs to keep. Must be in the range [0, 1].
      rate = 0.01
    }
    # The configuration for a drop field action.
    type = "SAMPLE"
  }

  rules {
    # Second rule
    name   = "Drop NGINX information logs"
    filter = "service = \"nginx\" and severity = \"INFO\" and httpRequest.status = \"200\""
    mode   = "ENABLED"
    type   = "DROP"
  }

  rules {
    # Third rule
    name = "Drop Kubernetes fields from development environments"

    drop_field {
      # Regular expression to match the field names to drop.
      field_regex = ".*"
      # Selector to indicate field path. Use 'parent[child]' syntax to
      # indicate nesting.
      parent_path {
        selector = "kubernetes"
      }
    }

    filter = "chronosphere_namespace = \"rc-logging\" and kubernetes.cluster_name = \"rc-b\""
    mode   = "ENABLED"
    type   = "DROP_FIELD"
  }

  rules {
    # Fourth rule
    name   = "error replacement"
    filter = "service = \"nginx\" and severity = \"ERROR\""
    mode   = "ENABLED"

    replace_field {
      field {
        selector = "error_code"
      }

      mapped_value {
        pairs {
          key   = "INTERNAL_SERVER_ERROR"
          value = "500"
        }

        pairs {
          key   = "NOT_FOUND"
          value = "401"
        }
      }

      replace_all   = true
      replace_mode  = "MAPPED_VALUE"
      replace_regex = ".*"
    }

    type = "REPLACE_FIELD"
  }

  rules {
    # Fifth rule
    name   = "Emit counter for NGINX errors"
    filter = "service = \"nginx\" and severity = \"ERROR\""
    mode   = "ENABLED"
    type   = "EMIT_METRICS"

    emit_metrics {
      # Prometheus-compatible metric name.
      name = "nginx_errors_total"
      mode = "COUNTER"

      labels {
        key = "service"
        value {
          selector = "service"
        }
      }

      labels {
        key = "error_code"
        value {
          selector = "error_code"
        }
      }
    }
  }
}

Edit control rules

When creating or editing control rules, you can use the Code Config tool to view code representations of the rules. The displayed code also responds to changes you make in the Visual editor tab. Select from the following methods to edit control rules.

Web
Chronoctl
Terraform
API

In the navigation menu, click Go to Admin and then select Control > Logs Ingest.
In the row of the control rule you want to edit, click the three vertical dots icon and then click Edit control rule.
In the Edit control rule drawer, make changes to your control rule, and then click Save.
Click the Code config tab and use the Code Config tool to apply the definition.

Observability Platform saves changes to your control rule.

To update control rules with Chronoctl, use the log-control-config update command:

chronoctl log-control-config update

Update the control rule definition file.
Run the following command to submit the changes:
```
chronoctl log-control-config update -f FILE_NAME.yaml
```
Replace FILE_NAME with the name of the YAML definition file you want to use.

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To edit control rules using Terraform:

Create or edit a Terraform file that updates the resource’s existing properties.
Run this command to apply the changes:
```
terraform apply
```

You can also Use the Code Config tool to view the control rule’s Terraform representation.

To complete this action with the Chronosphere API, use the UpdateLogControlConfig endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Reorder control rules

Control rules are applied in descending order. To change the order control rules are applied, reorder rules. Disabled rules are skipped.

Web
Chronoctl
Terraform
API

In the navigation menu, click Go to Admin and then select Control > Logs Ingest.
In the Control rules section, click Re-order rules.
In the Edit control rules drawer, in the row of the control rule you want to move, click the drag indicator icon and move the control rule to a different order position.
Click Save.
Click the Code config tab and use the Code Config tool to apply the definition.

Observability Platform saves changes to your control rules.

To reorder control rules with Chronoctl, use the log-control-config update command:

chronoctl log-control-config update

Update the control rule definition file.
Run the following command to submit the changes:
```
chronoctl log-control-config update -f FILE_NAME.yaml
```
Replace FILE_NAME with the name of the YAML definition file you want to use.

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To reorder control rules using Terraform:

Create or edit a Terraform file that updates the resource’s existing properties.
Run this command to apply the changes:
```
terraform apply
```

You can also use the Code Config tool to view the control rule’s Terraform representation.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Enable or disable control rules

Each rule has a mode, which can be enabled or disabled. By default, any created rule is enabled. Any rule that’s enabled is applied. Disabled rules are skipped.

Web
Chronoctl
Terraform
API

In the navigation menu, click Go to Admin and then select Control > Logs Ingest.
In the row of the control rule you want to enable or disable, click the toggle in the Enabled column.
Click Save.
Click the Code config tab and use the Code Config tool to apply the definition.

Observability Platform saves changes to your control rules.

To enable or disable control rules with Chronoctl, use the log-control-config update command:

chronoctl log-control-config update

In the control rule definition file, for the rule you want to modify, change the current mode. For example, to disable a rule, change mode: ENABLED to mode: DISABLED.
Run the following command to submit the changes:
```
chronoctl log-control-config update -f FILE_NAME.yaml
```
Replace FILE_NAME with the name of the YAML definition file you want to use.

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To enable or disable control rules using Terraform:

In the control rule Terraform file, change the current mode for the rule you want to modify. For example, to disable a rule, change its mode = ENABLED to mode = DISABLED.
Run this command to apply the changes:
```
terraform apply
```

You can also use the Code Config tool to view the control rule’s Terraform representation.

To complete this action with the Chronosphere API, use the UpdateLogControlConfig endpoint.In the control rule definition, change the current "mode" for the rule you want to modify. For example, to disable a rule, change its "mode": "ENABLED" to "mode": "DISABLED".Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Delete control rules

Select from the following methods to delete control rules.

Web
Chronoctl
Terraform
API

In the navigation menu, click Go to Admin and then select Control > Logs Ingest.
In the row of the control rule you want to delete, click the three vertical dots icon and then click Delete control rule.
Click the Code config tab and use the Code Config tool to apply the definition.

Observability Platform deletes your control rule.

To delete a control rule with Chronoctl, use the chronoctl log-control-config delete command:

chronoctl log-control-config delete

Users can modify Terraform-managed resources only by using Terraform. Learn more.

To delete a resource that’s managed by Terraform:

Edit your Terraform configuration file to remove the pre-existing resource definition.
Run this command to remove the resource from Observability Platform:
```
terraform apply
```

To complete this action with the Chronosphere API, use the DeleteLogControlConfig endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Observability Platform

Telemetry Pipeline

Tools

More information

Applying control rules

View control rules

Create control rules

Chronoctl example

Terraform example

Edit control rules

Reorder control rules

Enable or disable control rules

Delete control rules

Observability Platform

Telemetry Pipeline

Tools

More information

Documentation Index

​Applying control rules

​View control rules

​Create control rules

​Chronoctl example

​Terraform example

​Edit control rules

​Reorder control rules

​Enable or disable control rules

​Delete control rules

Applying control rules

View control rules

Create control rules

Chronoctl example

Terraform example

Edit control rules

Reorder control rules

Enable or disable control rules

Delete control rules