Log datasets

Understanding your license consumption helps identify where you’re spending the most money on your logging data, and which services or operations are consuming the most of your license capacity. After creating partitions, you can create log datasets, which are an optional control mechanism that let you map sets of logs to named groups relevant to your organization. Use log datasets to track processed and persisted bytes for those groups over time. For example, you might create a Shopper dataset based on data like services, customer IDs, and tags that relate to your shopping app. Viewing that dataset provides a snapshot of log data volume associated with the entire business unit related to your shopping app. After creating datasets, you can update your partitions to reference datasets instead of using standalone partitions. Use log datasets when you need reusable slices of data that are tracked consistently over time, especially for when you anticipate referencing the same complex filters across multiple partitions, queries, or controls.

View datasets

Select from the following methods to view and filter available datasets.

Chronoctl
API

To use Chronoctl to return all datasets including trace datasets (if enabled in your tenant), use the chronoctl datasets list command:

chronoctl datasets list

To return log datasets only, add the type argument to the command and specify LOGS:

chronoctl datasets list --type LOGS

To filter for a specific logs dataset, add the slugs argument to the command:

chronoctl datasets list --slugs SLUG

Replace SLUG with the slug of the dataset you want to display.

To complete this action with the Chronosphere API, use the ListDatasets endpoint.To return only log datasets, specify type=LOGS in the URL of your API call.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Create datasets

Use one of the following methods to create log datasets. Define and test your query in Logs Explorer, and then map that query to the resource you want to create. After creating a dataset, you can define a budget to allocate a percentage of your log license limit to each dataset. To create a dataset:

Define a query in Logs Explorer that represents the data you want included in the dataset. For example, the following query returns all logs for the shopper-service service in the production-us environment:
```
service = "shopper-service"
AND environment = "production-us"
```
After defining the underlying query, use one of the following methods to map the query to a dataset that represents the business unit you want to track log data for. See the Chronoctl YAML definition and the Terraform resource definition for examples.

Chronoctl
Terraform
API

If you don’t already have a YAML configuration file, use the scaffold Chronoctl parameter to generate a template for a specific resource type:

chronoctl datasets scaffold

You can redirect the results (using the redirection operator >) to a file for editing.

To create a dataset with Chronoctl:

Run the following command to generate a sample dataset configuration you can use as a template:
```
chronoctl datasets scaffold
```
In the template, kind: Dataset defines an individual dataset.
With a completed definition, submit it with:
```
chronoctl datasets create -f FILE_NAME
```
Replace FILE_NAME with the name of the YAML definition file you want to use.

See the Chronoctl dataset example for a completed dataset definition.

When you run terraform plan to generate an execution plan, Chronosphere automatically tests configurations that include notification policies by submitting them as dry runs. For details, see the Terraform provider documentation.

To create a dataset with Terraform:

Create or edit a Terraform file and add the definition by using the chronosphere_dataset type, followed by a name in a resource declaration.
Run this command to apply the changes:
```
terraform apply
```

See the Terraform dataset example for a completed dataset resource.

To complete this action with the Chronosphere API, use the CreateDataset endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Next steps

After creating a dataset, update partitions to reference the slug of your dataset. You can then define budgets, which are optional shaping policies attached to a partition. Budgets provide guardrails to avoid runaway usage and overspending, while providing the flexibility to enforce accountability at the right level of ownership.

Chronoctl dataset example

The following YAML definition consists of one dataset named Logs for payment service in production US. This dataset includes a service named shopper-service in the production-us environment. You must escape quotation marks in the query field. For example, to specify "shopper-service" as a value in your query, enter \"shopper-service\". The query field accepts both AND and OR operators, but only one of these operator types is permitted. You can use multiple instances of the same operator, such as multiple AND or multiple OR operators.

api_version: v1/config
kind: Dataset
spec:
  # Required name of the dataset. Can be modified after the dataset is created.
  name: Logs for payment service in production US
  # Unique identifier of the dataset. If not provided, a slug is generated based
  # on the name field. Can't be modified after the dataset is created.
  slug: log-shopping-service-production
  # Optional description for the dataset.
  description: Logs for payment service in US production environment
  # Defining characteristics of the dataset.
  configuration:
    # Dataset type, which must be LOGS.
    type: LOGS
    log_dataset:
        match_criteria:
        # Query to match logs. This query can include AND or OR operators, but
        # only one of these operator types is allowed in the query. You can use
        # multiple instances of the same operator, such as multiple AND or multiple
        # OR operators.
          query: "service = \"shopper-service\" AND environment = \"production-us\""

Terraform dataset example

The following Terraform resource creates a dataset that Terraform refers to as shopper_production_us, with a human-readable name of Logs for payment service in production US. This dataset includes a service named shopper-service in the production-us environment.

resource "chronosphere_dataset" "shopper_production_us" {
  name        = "Logs for payment service in production US"
  description = "Logs for payment service in US production environment"
  configuration {
    type = "LOGS"

    log_dataset {
      match_criteria {
        query = "service = 'shopper-service' AND environment = 'production-us'"
      }
    }
  }
}

Update datasets

Select from the following methods to update log datasets.

Chronoctl
Terraform
API

Update the datasets definition file.
Run the following command to submit the changes:
```
chronoctl datasets update -f FILE_NAME.yaml
```
Replace FILE_NAME with the name of the YAML definition file you want to use.

To edit datasets using Terraform:

Create or edit a Terraform file that updates the resource’s existing properties.
Run this command to apply the changes:
```
terraform apply
```

To complete this action with the Chronosphere API, use the UpdateDataset endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Delete datasets

Select from the following methods to delete log datasets.

Users can modify Terraform-managed resources only by using Terraform. Learn more.

Chronoctl
Terraform
API

To delete a dataset with Chronoctl, use the chronoctl datasets delete command:

chronoctl datasets delete SLUG

Replace SLUG with the slug of the dataset you want to delete.For example, to delete a dataset with the slug log-shopping-service-production:

chronoctl datasets delete log-shopping-service-production

To delete a resource that’s managed by Terraform:

Edit your Terraform configuration file to remove the pre-existing resource definition.
Run this command to remove the resource from Observability Platform:
```
terraform apply
```

To complete this action with the Chronosphere API, use the DeleteDataset endpoint.Because the Chronosphere API requires authentication, include an API token with your curl request, as shown in the following example. For more details, see Create an API token.

export CHRONOSPHERE_API_TOKEN="TOKEN"
export CHRONOSPHERE_DOMAIN="INSTANCE.chronosphere.io"

curl -H "API-Token: ${CHRONOSPHERE_API_TOKEN}" \
     -X METHOD "https://${CHRONOSPHERE_DOMAIN}/ENDPOINT_PATH"

Replace the following:

TOKEN: Your API token.
INSTANCE: The subdomain name for your organization’s Observability Platform instance.
METHOD: The HTTP method to use with the request, such as GET or POST.
ENDPOINT_PATH: The specific endpoint you want to access.

Observability Platform

Telemetry Pipeline

Tools

More information

View datasets

Create datasets

Next steps

Chronoctl dataset example

Terraform dataset example

Update datasets

Delete datasets

Observability Platform

Telemetry Pipeline

Tools

More information

​View datasets

​Create datasets

​Next steps

​Chronoctl dataset example

​Terraform dataset example

​Update datasets

​Delete datasets

View datasets

Create datasets

Next steps

Chronoctl dataset example

Terraform dataset example

Update datasets

Delete datasets