Recommended configuration

Recommended ingestion configuration

Ingesting log data is a key step to ensure you parse your log data correctly before that data enters Chronosphere.

Although you can use your existing ingestion pipeline, Chronosphere recommends using Calyptia Core Agent or Calyptia Core Agent plus Calyptia Core.

Calyptia Core Agent

You can run Calyptia Core Agent (opens in a new tab) as an agent that collects data from your app, parses that data, and sends the data directly to Chronosphere. Use this deployment method if:

  • You're comfortable managing YAML-based configuration files.
  • You plan on parsing data in Calyptia Core Agent, and don't want to add Calyptia Core as another component in your ingestion pipeline.

However, this method means you might need to write your own parser for complex configurations, whereas Calyptia Core has built-in parsers for managing complex configurations. See Configure Calyptia Core Agent for more information.

Configure Calyptia Core Agent

Calyptia Core Agent can be an agent that runs in your environment, a data collector, or serves both of these purposes. In this configuration, you run Calyptia Core Agent as an agent that collects data from your app, parses that data, and sends the data directly to Chronosphere.

Complete the following steps to ingest data with Calyptia Core Agent:

  1. Create a configuration file (opens in a new tab) to define your services.

    Alternatively, you can create a YAML configuration file (opens in a new tab). See this example (opens in a new tab) for reference.

  2. Optional: Add variables (opens in a new tab) or commands (opens in a new tab) to enhance your configuration file.

  3. Add inputs (opens in a new tab) to your configuration file.

  4. Create a parsers.conf configuration file to define which parser to use. The built-in parsers (opens in a new tab) cover most use cases.

  5. Optional: Add filters (opens in a new tab) to your configuration file to enrich your data.

  6. Define the output (opens in a new tab) destination for your data, which is your LogScale tenant. The full URL for LogScale is:

    https://chronosphere.oem-1.logscale.us-1.crowdstrike.com/

    See the example configuration file for more information.

Calyptia Core

While you can run Calyptia Core Agent on its own, you can also have it send data to Calyptia Core (opens in a new tab) to do your parsing there. Use this deployment method if:

  • You want a graphical interface to manage your agents and pipeline configurations, rather than using YAML-based configuration files.
  • You want the ability to run sample actions in the pipeline to preview your data transformations before applying the changes.

This method adds Calyptia Core as another component in your ingestion pipeline. However, previewing your transformations means you can safely modify the parsing logic in your pipeline before making changes to your data. See Configure Calyptia Core Agent plus Calyptia Core for more information.

Configure Calyptia Core Agent with Calyptia Core

In this configuration, you run Calyptia Core Agent as an agent in your environment that collects data and sends it to Calyptia Core for processing. You parse your data in a Calyptia Core pipeline, and then send the processed data to Chronosphere. You can manage your Calyptia Core Agent in Calyptia Core.

Complete the following steps in Calyptia Core Agent:

  1. Create a configuration file (opens in a new tab) to define your services.

    Alternatively, you can create a YAML configuration file (opens in a new tab). See this example (opens in a new tab) for reference.

  2. Add inputs (opens in a new tab) to your configuration file.

  3. Define the output (opens in a new tab) destination for your data, which is Calyptia Core.

Complete the following steps in Calyptia Core:

  1. Create an ingest pipeline (opens in a new tab) to read data from your application. You can transform, drop, and route data in a pipeline.
  2. Define a secret (opens in a new tab) for your pipeline.
  3. Optional: Define a parser (opens in a new tab) to determine which fields are extracted during ingest.
  4. Add processing rules (opens in a new tab) to your ingest pipeline.

Configuration file example

The main Calyptia configuration file supports these section types:

  • SERVICE
  • INPUT
  • FILTER
  • OUTPUT

A section can contain individual entries, which are defined by a line of text that contains both a key and a value.

See the Configuration file (opens in a new tab) page in the Calyptia Core Agent documentation for more information.

Use the following example as a model for creating your own configuration file. This example assumes your logs are formatted in JSON, as indicated by the /api/v1/ingest/json value for the URI key.

The LOGSCALE_INGEST_TOKEN is the LogScale ingest token you created when configuring LogScale.

[SERVICE]
    Daemon Off
    Flush 1
    Log_Level info
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_Port 2020
    Health_Check On
 
[INPUT]
    Name tail
    Path /var/log/containers/*.log
    multiline.parser docker, cri
    Tag kube.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On
 
[INPUT]
    Name systemd
    Tag host.*
    Systemd_Filter _SYSTEMD_UNIT=kubelet.service
    Read_From_Tail On
 
[FILTER]
    Name kubernetes
    Match kube.*
    Merge_Log On
    Merge_Log_Key On
    Keep_Log Off
    K8S-Logging.Parser On
    K8S-Logging.Exclude On
 
[OUTPUT]
    Name             chronosphere
    Match            *
    Host             chronosphere.oem-1.logscale.us-1.crowdstrike.com
    Port             443
    URI              /api/v1/ingest/json
    Header           Authorization Bearer LOGSCALE_INGEST_TOKEN
    tls              On
    tls.verify       On
    compress         On
    format           json
    json_date_key    @timestamp
    json_date_format iso8601