Log datasets

OBSERVABILITY PLATFORM

Log datasets

This feature is available only to specific Chronosphere Observability Platform users, and has not been announced or officially released. Do not share or discuss this feature, or information about it, with anyone outside of your organization.

Understanding your license consumption helps identify where you're spending the most money on your logging data, and which services or operations are consuming the most of your license capacity.

Log datasets are a control mechanism that let you map sets of logs to named groups relevant to your organization, and then track processed and persisted bytes for those groups over time.

For example, you might create a Shopper dataset based on data like services, customer IDs, and tags that relate to your shopping app. Viewing that dataset provides a snapshot of log data volume associated with the entire business unit related to your shopping app.

After creating datasets, you can create budgets to allocate a percentage of your license consumption percentage to each dataset. Chronosphere Observability Platform provides the Logging License Consumption dashboard so you can track your persisted log data volume over time to identify which of your datasets are consuming the most of your license capacity.

View datasets

Select from the following methods to view and filter available datasets.

To use Chronoctl to return all datasets including trace datasets (if enabled in your tenant), use the chronoctl datasets list command:

chronoctl datasets list

To return log datasets only, add the type argument to the command and specify LOGS:

chronoctl datasets list --type LOGS

To filter for a specific logs dataset, add the slugs argument to the command:

chronoctl datasets list --slugs SLUG

Replace SLUG with the slug of the dataset you want to display.

Create datasets

Use one of the following methods to create log datasets. Define and test your query in Logs Explorer, and then map that query to the resource you want to create.

After creating a dataset, you can define a budget to allocate a percentage of your log license limit to each dataset.

To create a dataset:

Define a query in Logs Explorer that represents the data you want included in the dataset. For example, the following query returns all logs for the shopper-service service in the production-us environment:
```
service = "shopper-service"
AND environment = "production-us"
```
After defining the underlying query, use one of the following methods to map the query to a dataset that represents the business unit you want to track log data for.

See the Chronoctl YAML definition and the Terraform resource definition for examples.

The scaffold parameter requires Chronoctl version 0.55 or later.

If you don't already have a YAML configuration file, use the scaffold Chronoctl parameter to generate a template for a specific resource type:

chronoctl datasets scaffold

You can redirect the results (using the redirection operator >) to a file for editing.

To create a dataset with Chronoctl:

Run the following command to generate a sample dataset configuration you can use as a template:
```
chronoctl datasets scaffold
```
In the template, kind: Dataset defines an individual dataset.
With a completed definition, submit it with:
```
chronoctl datasets create -f FILE_NAME
```
Replace FILE_NAME with the name of the YAML definition file you want to use.

See the Chronoctl dataset example for a completed dataset definition.

Chronoctl dataset example

The following YAML definition consists of one dataset named Logs for payment service in production US. This dataset includes a service named shopper-service in the production-us environment.

You must escape quotation marks in the query field. For example, to specify "shopper-service" as a value in your query, enter \"shopper-service\". The query field accepts both AND and OR operators, but only one of these operator types is permitted. You can use multiple instances of the same operator, such as multiple AND or multiple OR operators.

api_version: v1/config
kind: Dataset
spec:
  # Required name of the dataset. Can be modified after the dataset is created.
  name: Logs for payment service in production US
  # Unique identifier of the dataset. If not provided, a slug is generated based
  # on the name field. Can't be modified after the dataset is created.
  slug: log-shopping-service-production
  # Optional description for the dataset.
  description: Logs for payment service in US production environment
  # Defining characteristics of the dataset.
  configuration:
    # Dataset type, which must be LOGS.
    type: LOGS
    log_dataset:
        match_criteria:
        # Query to match logs. This query can include AND or OR operators, but
        # only one of these operator types is allowed in the query. You can use
        # multiple instances of the same operator, such as multiple AND or multiple
        # OR operators.
          query: "service = \"shopper-service\" AND environment = \"production-us\""

Terraform dataset example

The following Terraform resource creates a dataset that Terraform refers to as shopper_production_us, with a human-readable name of Logs for payment service in production US.

This dataset includes a service named shopper-service in the production-us environment.

resource "chronosphere_dataset" "shopper_production_us" {
  name        = "Logs for payment service in production US"
  description = "Logs for payment service in US production environment"
  configuration {
    type = "LOGS"
 
    log_dataset {
      match_criteria {
        query = "service = 'shopper-service' AND environment = 'production-us'"
      }
    }
  }
}

Delete datasets

Select from the following methods to delete log datasets.

Users cannot modify Terraform-managed resources in the user interface, with Chronoctl, or by using the API. Learn more.

To delete a dataset with Chronoctl, use the chronoctl datasets delete command:

chronoctl datasets delete SLUG

Replace SLUG with the slug of the dataset you want to delete.

For example, to delete a dataset with the slug log-shopping-service-production:

chronoctl datasets delete log-shopping-service-production

External tasks Live Telemetry Analyzer