Recommended ingestion configuration

Ingesting log data is a key step to ensure you parse your log data correctly before that data enters Chronosphere Observability Platform.

Although you can use your existing ingestion pipeline, Chronosphere recommends using Core Agent or your existing logging processes in conjunction with Core Agent and Telemetry Pipeline.

To get started with ingesting logs using Telemetry Pipeline:

1. Create an ingest API token.
2. Choose which configuration you want to ingest log data:
   • Core Agent
   • Telemetry Pipeline

Create an ingest API token

Before ingesting log data, you need to create an API token to authenticate with and ingest data to Observability Platform. See Ingest tokens (opens in a new tab) in the LogScale documentation for more information about creating ingest tokens.

To create an ingest API token, you must have administrative permissions.

1. In the navigation menu select Explorers > Logs Explorer.
2. Click Repository settings to open LogScale repository settings in a new tab.
3. In the LogScale interface, select the repository you want to create an API token for.
4. In the main LogScale navigation, click Settings.
5. In the Ingest section of the sidebar navigation, click Ingest tokens.
6. Click Add token to create a new ingest token.
7. Enter a name for your token.
8. Optional: Assign a parser for your token if you want to parse data during ingestion.
9. Click Save to save your ingest token.

Use this ingest API token in your Telemetry Pipeline configuration to authenticate with and begin sending log data to Observability Platform.

Core Agent

You can run Core Agent (opens in a new tab) as an agent that collects data from your app, parses that data, and sends the data directly to Observability Platform. Use this deployment method if:

• You're comfortable managing YAML-based configuration files.
• You plan on parsing data in Core Agent, and don't want to add Telemetry Pipeline as another component in your ingestion pipeline.

However, this method means you might need to write your own parser for complex configurations, whereas Telemetry Pipeline has built-in parsers for managing complex configurations. See Configure Core Agent for more information.

Configure Core Agent

Core Agent can be an agent that runs in your environment, a data collector, or serves both of these purposes. In this configuration, you run Core Agent as an agent that collects data from your app, parses that data, and sends the data directly to Observability Platform.

Complete the following steps to ingest data with Core Agent:

1. Create a YAML configuration file (opens in a new tab).

2. Add inputs (opens in a new tab) to your configuration file.

3. Add a processors collection to your configuration file to define which parsers to apply data transformations and filtering to incoming data records before processing data further in the pipeline. See Processors (opens in a new tab) for more information.

4. Optional: Add filters (opens in a new tab) to your configuration file to enrich your data.

5. Define the output (opens in a new tab) destination for your data, which is your Observability Platform tenant. The full URL is:

   https://ADDRESS.ingest.logs.chronosphere.io/

   Replace ADDRESS with your company name prefixed to your Observability Platform instance that ends in ingest.logs.chronosphere.io. For example, MY_COMPANYingest.logs.chronosphere.io.

Telemetry Pipeline

While you can run Core Agent on its own, you can also combine it with your existing logging processes to send data to Telemetry Pipeline to do your parsing there. Use this deployment method if:

• You want a graphical interface to manage your agents and pipeline configurations, rather than using YAML-based configuration files.
• You want the ability to run sample actions in the pipeline to preview your data transformations before applying the changes.

This method adds Telemetry Pipeline as another component in your ingestion pipeline. However, previewing your transformations means you can safely modify the parsing logic in your pipeline before making changes to your data. See Configure Core Agent plus Telemetry Pipeline for more information.

Configure Core Agent with Telemetry Pipeline

In this configuration, you run Core Agent as an agent in your environment that collects data and sends it to Telemetry Pipeline for processing. You parse your data in Telemetry Pipeline, and then send the processed data to Observability Platform. You can manage your Core Agent in Telemetry Pipeline.

Core Agent configuration

Complete the following steps in Core Agent:

1. Create a YAML configuration file (opens in a new tab).
2. Add inputs (opens in a new tab) to your configuration file.
3. Define the output (opens in a new tab) destination for your data, which is Telemetry Pipeline.

Telemetry Pipeline configuration

Complete the following steps in Telemetry Pipeline:

1. Create an ingest pipeline to read data from your application. You can transform, drop, and route data in a pipeline.
2. Define a secret for your pipeline.
3. Optional: Define a parser to determine which fields are extracted during ingest.
4. Add processing rules to your ingest pipeline.