OpenTelemetry Collector

Ingest logs using the OpenTelemetry Collector

The OpenTelemetry Collector (opens in a new tab) is vendor-agnostic, open source, and supports popular backends and the OpenTelemetry Protocol. You can use the OpenTelemetry Collector to ingest log data to Chronosphere Observability Platform.

Get started

To get started with the OpenTelemetry Collector:

Instrument your app (opens in a new tab) with an OpenTelemetry SDK (opens in a new tab).
Create a repository ingest token (opens in a new tab) to authenticate with LogScale and ingest data.

You can then configure your OpenTelemetry Collector to ingest log data.

Configure your OpenTelemetry Collector

To send log data to Observability Platform, configure the OpenTelemetry Collector (opens in a new tab). You must send all traffic over HTTPS.

To configure the OpenTelemetry Collector to ingest log data:

Pull the OpenTelemetry Docker image (opens in a new tab) to run the OpenTelemetry Collector in a Docker container:
```
docker pull otel/opentelemetry-collector-contrib:VERSION
docker run otel/opentelemetry-collector-contrib:VERSION
```
Replace VERSION with the version of the OpenTelemetry Collector you want to run, which must be version 0.83 or later.
In the OpenTelemetry Collector config.yaml file (opens in a new tab), apply the following settings to modify the exporter YAML collection. Specify an endpoint that points to your Observability Platform tenant, and include the LogScale ingest token you created as an HTTP header.

You can modify the OpenTelemetry Collector configuration (opens in a new tab) if you want to change the defaults. For example, you can reference a config.yaml in a different location, such as a Kubernetes ConfigMap.

Use the otlphttp exporter to send log data, which differs from the otlp exporter used to send metric and trace data to Chronosphere.
```
exporters:
  otlphttp/logging-chronosphere:
    endpoint: https://ORGANIZATION.ingest.logs.chronosphere.io/api/v1/ingest/otlp
    retry_on_failure:
      enabled: false
    compression: gzip
    headers:
      Authorization: "Bearer $API_TOKEN"
```
- ORGANIZATION: Your organization name prefixed to your Observability Platform instance that ends in .ingest.logs.chronosphere.io/api/v1/ingest/otlp. For example, if your organization name is acme, the resulting endpoint URL is http://acme.ingest.logs.chronosphere.io/api/v1/ingest/otlp.
- API_TOKEN: The API token you created to authenticate with LogScale, specified as an HTTP header. Chronosphere recommends calling this value as an environment variable. You can find ingest tokens on the LogScale repository settings (opens in a new tab) page.
- retry_on_failure: Set to false. The Chronosphere OTLP ingest API doesn't signal retryable errors.
  
  The Chronosphere OpenTelemetry endpoint supports gzip compression only.
Configure batch processing (opens in a new tab). Sending telemetry in batches improves data compression and reduces the number of outgoing connections required to transmit the data.

For example, the following configuration enforces a maximum batch size limit of 2000 logs without introducing any artificial delays:
```
processors:
  batch:
    timeout: 1s
    send_batch_size: 1000
    send_batch_max_size: 2000
```
The timeout, batch size, and batch max size are default recommendations. Monitor the exporter send and enqueueing failure metrics to tune these parameters based on your workload.

Instruct the OpenTelemetry Collector to load the API token from an environment variable.

⚠️

Never share or store your API token in plain text. Chronosphere recommends using tools like SOPs to securely store this information.

   # In the OpenTelemetry Collector Deployment:
     - name: $API_TOKEN
       valueFrom:
         secretKeyRef:
           name: chronosphere-api-token
           key: apiToken
 
   # The accompanying Secret:
     apiVersion: v1
     kind: Secret
     metadata:
       name: chronosphere-api-token
       namespace: YOUR_NAMESPACE
     type: Opaque
     data:
       apiToken: $API_TOKEN

Save your OpenTelemetry Collector config.yaml file.

Observability Platform should begin ingesting log data.

Parse your data

After configuring your OpenTelemetry Collector to ingest log data in Chronosphere, you can use the LogScale Query Language (opens in a new tab) to create a parser that defines how to create fields from your raw log data. Chronosphere recommends using the following example parser for parsing OpenTelemetry log data. For more information about creating LogScale parsers, see Creating a parser (opens in a new tab) in the LogScale documentation.

To create a parser for OpenTelemetry data:

In the navigation menu select Explorers > Logs Explorer.
Click Repository settings to open the full LogScale UI.
In LogScale, click the repository you want to create a parser for.
In the top navigation, click Parsers, and then click New parser.
Enter a name for your parser, such as otel, click Empty parser, and then click Create.

The LogScale UI opens the parsing editor, where you define your parser.

In the LogScale parsing editor, enter the following parser, which conditionally parses the timestamp based on the present field. Alternatively, you can copy the following YAML example, save it with a .yaml extension, and upload it directly to LogScale as a parser template.

parseJson()
// The parser considers four timestamp variations: the `timeUnixNano` and
// `observedTimeUnixNano` fields in the official OpenTelemetry Protocol (OTLP)
// specification, and the `timeUnix` and `observedTimeUnix` object fields that are
// often used in OTLP messages.
| case {
     // Look for the Timestamp field first. This should map to
     // the time that the logRecord was created.
     //
     // Use the official scopeLog.logRecord.timeUnixNano, if it exists.
     scopeLog.logRecord.timeUnixNano = * AND scopeLog.logRecord.timeUnixNano > 0
| parseTimestamp(field=scopeLog.logRecord.timeUnixNano, format="nanos");

     // Look for the scopeLog.logRecord.timeUnix.millis as a fallback.
     scopeLog.logRecord.timeUnix.millis = * AND scopeLog.logRecord.timeUnix.millis > 0
| parseTimestamp(field=scopeLog.logRecord.timeUnix.millis, format="unixTimeMillis");

     // If the Timestamp field doesn't exist, use ObservedTimestamp instead,
     // which should map to the time that the logRecord was seen by
     // a collector. The time could be equivlant to the creation time if logs
     // are sent directly from an app.
     //
     // Use the official scopeLog.logRecord.observedTimeUnixNano.
     scopeLog.logRecord.observedTimeUnixNano = * AND scopeLog.logRecord.observedTimeUnixNano > 0
| parseTimestamp(field=scopeLog.logRecord.observedTimeUnixNano, format="nanos");

     // Look for scopeLog.logRecord.observedTimeUnix.millis as a fallback.
     scopeLog.logRecord.observedTimeUnix.millis = * AND scopeLog.logRecord.observedTimeUnix.millis > 0
| parseTimestamp(field=scopeLog.logRecord.observedTimeUnix.millis, format="unixTimeMillis");

     // If none of the above fields exist, use the ingest timestamp.
    * }

Click Save to save your parser.

YAML parser example

If you want to upload a YAML file instead of creating a parser using the LogScale Query Language, copy the following example, save it with a .yaml extension, and then upload the file to LogScale as a parser template.

name: otel-logs
fieldsToBeRemovedBeforeParsing: []
testCases:
- assertions:
  - fieldsHaveValues:
      '@timestamp': '1715259247295'
    fieldsNotPresent: null
    outputEventIndex: 0
  event:
    rawString: "{\n   \"schemaUrl\":\"\",\n   \"resource\":{\n      \"attributes\"\
      :{\n         \"service.name\":\"quoteservice\",\n         \"service.instance.id\"\
      :\"ee8c82eb-38eb-4ce2-933f-68987fdf3769\",\n         \"service.namespace\":\"\
      otel-demo\",\n         \"k8s.namespace.name\":\"otel-demo\",\n         \"k8s.node.name\"\
      :\"gke-se-demo-data-gen-primary-1-a8e1c613-yjdu\",\n         \"k8s.pod.name\"\
      :\"store-quoteservice-6756b94867-kf44k\",\n         \"container.id\":\"c5dd4651fc78cd45097291684d6b3e194f56bb36f336813f6aee553450f8ee1e\"\
      ,\n         \"service.version\":\"1.0.0+no-version-set\",\n         \"telemetry.sdk.name\"\
      :\"opentelemetry\",\n         \"telemetry.sdk.language\":\"php\",\n        \
      \ \"telemetry.sdk.version\":\"1.0.0\",\n         \"telemetry.distro.name\":\"\
      opentelemetry-php-instrumentation\",\n         \"telemetry.distro.version\"\
      :\"1.0.0\",\n         \"process.runtime.name\":\"cli\",\n         \"process.runtime.version\"\
      :\"8.2.13\",\n         \"process.pid\":7,\n         \"process.executable.path\"\
      :\"/usr/local/bin/php\",\n         \"process.command\":\"public/index.php\"\
      ,\n         \"process.command_args\":[\n            \"public/index.php\"\n \
      \        ],\n         \"process.owner\":\"www-data\",\n         \"os.type\"\
      :\"linux\",\n         \"os.description\":\"5.15.146+\",\n         \"os.name\"\
      :\"Linux\",\n         \"os.version\":\"#1 SMP Sat Feb 17 13:12:02 UTC 2024\"\
      ,\n         \"host.name\":\"store-quoteservice-6756b94867-kf44k\",\n       \
      \  \"host.arch\":\"x86_64\",\n         \"k8s.pod.ip\":\"10.1.0.27\",\n     \
      \    \"k8s.pod.start_time\":\"2024-04-21T17:09:49Z\",\n         \"k8s.pod.uid\"\
      :\"ee8c82eb-38eb-4ce2-933f-68987fdf3769\",\n         \"k8s.deployment.name\"\
      :\"store-quoteservice\"\n      },\n      \"droppedAttributesCount\":0\n   },\n\
      \   \"scopeLog\":{\n      \"schemaUrl\":\"\",\n      \"scope\":{\n         \"\
      name\":\"slim-app\",\n         \"version\":\"\",\n         \"droppedAttributesCount\"\
      :0,\n         \"attributes\":{\n            \n         }\n      },\n      \"\
      logRecord\":{\n         \"spanId\":\"66CB580C7F0446D6\",\n         \"traceId\"\
      :\"1EF0DA8C9D0294FF87526A091CF5EA0D\",\n         \"severityText\":\"INFO\",\n\
      \         \"severityNumber\":\"SEVERITY_NUMBER_INFO\",\n         \"flags\":1,\n\
      \         \"droppedAttributesCount\":0,\n         \"body\":\"Calculated quote\"\
      ,\n         \"attributes\":{\n            \"context\":{\n               \"total\"\
      :35.6\n            },\n            \"environment\":\"prod\",\n            \"\
      az\":\"us-east-1\"\n         },\n         \"observedTimeUnixNano\": \"1715259247295000000\"\
      ,\n         \"timeUnixNano\": \"0\"\n      }\n   }\n}"
- assertions:
  - fieldsHaveValues:
      '@timestamp': '1715259247294'
    fieldsNotPresent: null
    outputEventIndex: 0
  event:
    rawString: "{\n   \"schemaUrl\":\"\",\n   \"resource\":{\n      \"attributes\"\
      :{\n         \"service.name\":\"quoteservice\",\n         \"service.instance.id\"\
      :\"ee8c82eb-38eb-4ce2-933f-68987fdf3769\",\n         \"service.namespace\":\"\
      otel-demo\",\n         \"k8s.namespace.name\":\"otel-demo\",\n         \"k8s.node.name\"\
      :\"gke-se-demo-data-gen-primary-1-a8e1c613-yjdu\",\n         \"k8s.pod.name\"\
      :\"store-quoteservice-6756b94867-kf44k\",\n         \"container.id\":\"c5dd4651fc78cd45097291684d6b3e194f56bb36f336813f6aee553450f8ee1e\"\
      ,\n         \"service.version\":\"1.0.0+no-version-set\",\n         \"telemetry.sdk.name\"\
      :\"opentelemetry\",\n         \"telemetry.sdk.language\":\"php\",\n        \
      \ \"telemetry.sdk.version\":\"1.0.0\",\n         \"telemetry.distro.name\":\"\
      opentelemetry-php-instrumentation\",\n         \"telemetry.distro.version\"\
      :\"1.0.0\",\n         \"process.runtime.name\":\"cli\",\n         \"process.runtime.version\"\
      :\"8.2.13\",\n         \"process.pid\":7,\n         \"process.executable.path\"\
      :\"/usr/local/bin/php\",\n         \"process.command\":\"public/index.php\"\
      ,\n         \"process.command_args\":[\n            \"public/index.php\"\n \
      \        ],\n         \"process.owner\":\"www-data\",\n         \"os.type\"\
      :\"linux\",\n         \"os.description\":\"5.15.146+\",\n         \"os.name\"\
      :\"Linux\",\n         \"os.version\":\"#1 SMP Sat Feb 17 13:12:02 UTC 2024\"\
      ,\n         \"host.name\":\"store-quoteservice-6756b94867-kf44k\",\n       \
      \  \"host.arch\":\"x86_64\",\n         \"k8s.pod.ip\":\"10.1.0.27\",\n     \
      \    \"k8s.pod.start_time\":\"2024-04-21T17:09:49Z\",\n         \"k8s.pod.uid\"\
      :\"ee8c82eb-38eb-4ce2-933f-68987fdf3769\",\n         \"k8s.deployment.name\"\
      :\"store-quoteservice\"\n      },\n      \"droppedAttributesCount\":0\n   },\n\
      \   \"scopeLog\":{\n      \"schemaUrl\":\"\",\n      \"scope\":{\n         \"\
      name\":\"slim-app\",\n         \"version\":\"\",\n         \"droppedAttributesCount\"\
      :0,\n         \"attributes\":{\n            \n         }\n      },\n      \"\
      logRecord\":{\n         \"spanId\":\"66CB580C7F0446D6\",\n         \"traceId\"\
      :\"1EF0DA8C9D0294FF87526A091CF5EA0D\",\n         \"severityText\":\"INFO\",\n\
      \         \"severityNumber\":\"SEVERITY_NUMBER_INFO\",\n         \"flags\":1,\n\
      \         \"droppedAttributesCount\":0,\n         \"body\":\"Calculated quote\"\
      ,\n         \"attributes\":{\n            \"context\":{\n               \"total\"\
      :35.6\n            },\n            \"environment\":\"prod\",\n            \"\
      az\":\"us-east-1\"\n         },\n         \"observedTimeUnixNano\": \"1715259247295000000\"\
      ,\n         \"timeUnixNano\": \"1715259247294000000\"\n      }\n   }\n}"
- assertions:
  - fieldsHaveValues:
      '@timestamp': '1715259247292'
    fieldsNotPresent: null
    outputEventIndex: 0
  event:
    rawString: "{\n   \"schemaUrl\":\"\",\n   \"resource\":{\n      \"attributes\"\
      :{\n         \"service.name\":\"quoteservice\",\n         \"service.instance.id\"\
      :\"ee8c82eb-38eb-4ce2-933f-68987fdf3769\",\n         \"service.namespace\":\"\
      otel-demo\",\n         \"k8s.namespace.name\":\"otel-demo\",\n         \"k8s.node.name\"\
      :\"gke-se-demo-data-gen-primary-1-a8e1c613-yjdu\",\n         \"k8s.pod.name\"\
      :\"store-quoteservice-6756b94867-kf44k\",\n         \"container.id\":\"c5dd4651fc78cd45097291684d6b3e194f56bb36f336813f6aee553450f8ee1e\"\
      ,\n         \"service.version\":\"1.0.0+no-version-set\",\n         \"telemetry.sdk.name\"\
      :\"opentelemetry\",\n         \"telemetry.sdk.language\":\"php\",\n        \
      \ \"telemetry.sdk.version\":\"1.0.0\",\n         \"telemetry.distro.name\":\"\
      opentelemetry-php-instrumentation\",\n         \"telemetry.distro.version\"\
      :\"1.0.0\",\n         \"process.runtime.name\":\"cli\",\n         \"process.runtime.version\"\
      :\"8.2.13\",\n         \"process.pid\":7,\n         \"process.executable.path\"\
      :\"/usr/local/bin/php\",\n         \"process.command\":\"public/index.php\"\
      ,\n         \"process.command_args\":[\n            \"public/index.php\"\n \
      \        ],\n         \"process.owner\":\"www-data\",\n         \"os.type\"\
      :\"linux\",\n         \"os.description\":\"5.15.146+\",\n         \"os.name\"\
      :\"Linux\",\n         \"os.version\":\"#1 SMP Sat Feb 17 13:12:02 UTC 2024\"\
      ,\n         \"host.name\":\"store-quoteservice-6756b94867-kf44k\",\n       \
      \  \"host.arch\":\"x86_64\",\n         \"k8s.pod.ip\":\"10.1.0.27\",\n     \
      \    \"k8s.pod.start_time\":\"2024-04-21T17:09:49Z\",\n         \"k8s.pod.uid\"\
      :\"ee8c82eb-38eb-4ce2-933f-68987fdf3769\",\n         \"k8s.deployment.name\"\
      :\"store-quoteservice\"\n      },\n      \"droppedAttributesCount\":0\n   },\n\
      \   \"scopeLog\":{\n      \"schemaUrl\":\"\",\n      \"scope\":{\n         \"\
      name\":\"slim-app\",\n         \"version\":\"\",\n         \"droppedAttributesCount\"\
      :0,\n         \"attributes\":{\n            \n         }\n      },\n      \"\
      logRecord\":{\n         \"spanId\":\"66CB580C7F0446D6\",\n         \"traceId\"\
      :\"1EF0DA8C9D0294FF87526A091CF5EA0D\",\n         \"severityText\":\"INFO\",\n\
      \         \"severityNumber\":\"SEVERITY_NUMBER_INFO\",\n         \"flags\":1,\n\
      \         \"droppedAttributesCount\":0,\n         \"body\":\"Calculated quote\"\
      ,\n         \"attributes\":{\n            \"context\":{\n               \"total\"\
      :35.6\n            },\n            \"environment\":\"prod\",\n            \"\
      az\":\"us-east-1\"\n         },\n         \"observedTimeUnix\":{\n         \
      \   \"millis\":1715259247292,\n            \"nanos\":604928\n         },\n \
      \        \"timeUnix\":{\n            \"millis\":0,\n            \"nanos\":0\n\
      \         }\n      }\n   }\n}"
- assertions:
  - fieldsHaveValues:
      '@timestamp': '1715259247293'
    fieldsNotPresent: null
    outputEventIndex: 0
  event:
    rawString: "{\n   \"schemaUrl\":\"\",\n   \"resource\":{\n      \"attributes\"\
      :{\n         \"service.name\":\"quoteservice\",\n         \"service.instance.id\"\
      :\"ee8c82eb-38eb-4ce2-933f-68987fdf3769\",\n         \"service.namespace\":\"\
      otel-demo\",\n         \"k8s.namespace.name\":\"otel-demo\",\n         \"k8s.node.name\"\
      :\"gke-se-demo-data-gen-primary-1-a8e1c613-yjdu\",\n         \"k8s.pod.name\"\
      :\"store-quoteservice-6756b94867-kf44k\",\n         \"container.id\":\"c5dd4651fc78cd45097291684d6b3e194f56bb36f336813f6aee553450f8ee1e\"\
      ,\n         \"service.version\":\"1.0.0+no-version-set\",\n         \"telemetry.sdk.name\"\
      :\"opentelemetry\",\n         \"telemetry.sdk.language\":\"php\",\n        \
      \ \"telemetry.sdk.version\":\"1.0.0\",\n         \"telemetry.distro.name\":\"\
      opentelemetry-php-instrumentation\",\n         \"telemetry.distro.version\"\
      :\"1.0.0\",\n         \"process.runtime.name\":\"cli\",\n         \"process.runtime.version\"\
      :\"8.2.13\",\n         \"process.pid\":7,\n         \"process.executable.path\"\
      :\"/usr/local/bin/php\",\n         \"process.command\":\"public/index.php\"\
      ,\n         \"process.command_args\":[\n            \"public/index.php\"\n \
      \        ],\n         \"process.owner\":\"www-data\",\n         \"os.type\"\
      :\"linux\",\n         \"os.description\":\"5.15.146+\",\n         \"os.name\"\
      :\"Linux\",\n         \"os.version\":\"#1 SMP Sat Feb 17 13:12:02 UTC 2024\"\
      ,\n         \"host.name\":\"store-quoteservice-6756b94867-kf44k\",\n       \
      \  \"host.arch\":\"x86_64\",\n         \"k8s.pod.ip\":\"10.1.0.27\",\n     \
      \    \"k8s.pod.start_time\":\"2024-04-21T17:09:49Z\",\n         \"k8s.pod.uid\"\
      :\"ee8c82eb-38eb-4ce2-933f-68987fdf3769\",\n         \"k8s.deployment.name\"\
      :\"store-quoteservice\"\n      },\n      \"droppedAttributesCount\":0\n   },\n\
      \   \"scopeLog\":{\n      \"schemaUrl\":\"\",\n      \"scope\":{\n         \"\
      name\":\"slim-app\",\n         \"version\":\"\",\n         \"droppedAttributesCount\"\
      :0,\n         \"attributes\":{\n            \n         }\n      },\n      \"\
      logRecord\":{\n         \"spanId\":\"66CB580C7F0446D6\",\n         \"traceId\"\
      :\"1EF0DA8C9D0294FF87526A091CF5EA0D\",\n         \"severityText\":\"INFO\",\n\
      \         \"severityNumber\":\"SEVERITY_NUMBER_INFO\",\n         \"flags\":1,\n\
      \         \"droppedAttributesCount\":0,\n         \"body\":\"Calculated quote\"\
      ,\n         \"attributes\":{\n            \"context\":{\n               \"total\"\
      :35.6\n            },\n            \"environment\":\"prod\",\n            \"\
      az\":\"us-east-1\"\n         },\n         \"observedTimeUnix\":{\n         \
      \   \"millis\":1715259247292,\n            \"nanos\":604928\n         },\n \
      \        \"timeUnix\":{\n            \"millis\":1715259247293,\n           \
      \ \"nanos\":568000\n         }\n      }\n   }\n}"
$schema: https://schemas.humio.com/parser/v0.3.0
script: |-
  parseJson()
  | case { // Conditionally parse the timestamp based on the present field.
          // The parser considers four timestamp variations: the `timeUnixNano` and
          // `observedTimeUnixNano` fields in the official OpenTelemetry Protocol (OTLP)
          // specification, and the `timeUnix` and `observedTimeUnix` object fields that are
          // often used in OTLP messages.
 
          // Look for the Timestamp field first. This should map to the time that
          // the logRecord was created at.
          //
          // Use the official scopeLog.logRecord.timeUnixNano if it exists.
          scopeLog.logRecord.timeUnixNano = * AND scopeLog.logRecord.timeUnixNano > 0
  | parseTimestamp(field=scopeLog.logRecord.timeUnixNano, format="nanos");
 
          // Look for the scopeLog.logRecord.timeUnix.millis as a fallback.
          scopeLog.logRecord.timeUnix.millis = * AND scopeLog.logRecord.timeUnix.millis > 0
  | parseTimestamp(field=scopeLog.logRecord.timeUnix.millis, format="unixTimeMillis");
 
          // If the Timestamp field doesn't exist, use ObservedTimestamp instead,
          // which should map to the time that the logRecord was seen by
          // a collector. The time could be equivlant to the creation time if logs
          // are sent directly from an app.
          //
          // Use the official scopeLog.logRecord.observedTimeUnixNano.
          scopeLog.logRecord.observedTimeUnixNano = * AND scopeLog.logRecord.observedTimeUnixNano > 0
  | parseTimestamp(field=scopeLog.logRecord.observedTimeUnixNano, format="nanos");
 
          // Look for scopeLog.logRecord.observedTimeUnix.millis as a fallback.
          scopeLog.logRecord.observedTimeUnix.millis = * AND scopeLog.logRecord.observedTimeUnix.millis > 0
  | parseTimestamp(field=scopeLog.logRecord.observedTimeUnix.millis, format="unixTimeMillis");
 
          // If none of the above fields exist, use the ingest timestamp.
         * }
tagFields: []

Existing configuration Data limits