Use Prometheus Agent to send metrics

This feature isn’t available to all Chronosphere Observability Platform users and might not be visible in your app. For information about enabling this feature in your environment, contact Chronosphere Support.

You can configure Prometheus to send metrics to Chronosphere Observability Platform using the Remote-Write Protocol 2.0 API over HTTPS.

Observability Platform supports only Prometheus Remote-Write Protocol versions 2.0.0 or later and Prometheus server 3.0.0 or later. All requests must be encrypted using TLS, compressed payloads should be smaller than 16 MB, and metric payloads must contain fewer than 10,000 items.

Prerequisites

To forward metrics from Prometheus, you must first create a restricted service account in Observability Platform, which generates an API token for Prometheus to authenticate with Observability Platform. The API token only needs write access for Prometheus to send metrics to Observability Platform.

⚠️

Store your API token in a secure location, such as a separate file or as a Kubernetes Secret, and reference it using an environment variable, such as $API_TOKEN. You can also bind a Kubernetes Secret (opens in a new tab) to an environment variable on the Pod.

If you lose your token, you must create a new service account.

Configure Prometheus

To implement remote write, you must configure your application and Prometheus to use it. You must also install Prometheus Agent, which is beyond the scope of these instructions and can vary depending on your usage.

For more information, see the Prometheus Operator documentation for installing the Prometheus Agent in Kubernetes (opens in a new tab) or the Prometheus installation documentation (opens in a new tab) for other scenarios.

Instrument your app with Prometheus client libraries (opens in a new tab) for scraping or push metrics (opens in a new tab) to the PushGateway.
Configure third-party exporters (opens in a new tab) to scrape or push metrics from third-party tools and services.
Add a remote_write (opens in a new tab) section to your prometheus.yml to forward metrics to Chronosphere.
```
remote_write:
  - url: https://<COMPANY>.chronosphere.io/data/api/v1/prom/remote/write
    remote_timeout: 30s
    protobuf_message: io.prometheus.write.v2.Request
    headers:
      API-Token: $API_TOKEN
      Chronosphere-Metrics-Validation-Response: SHORT
 
    queue_config:
      batch_send_deadline: 1s
      max_samples_per_send: 2000
```
- url: Replace <COMPANY> with the name of your company as it appears in the subdomain of the Observability Platform URL.
- remote_timeout: Defines the timeout for Remote Write requests before failing and retrying. Default: 30s.
- protobuf_message: Specifies the Remote-Write v2 protobuf format, which provides additional metric types and more efficient egress traffic than the v1 format. For more information, see the Prometheus Remote-Write 2.0 Specification (opens in a new tab).
- headers: HTTP headers required in each request.
  - API-Token: Provides an Observability Platform API token for a restricted service account with write permissions. See Prerequisites for more information.
  - Chronosphere-Metrics-Validation-Response: Controls the verbosity of details in the response body for partial writes that describe why invalid series were rejected. The Prometheus server logs partial-write responses for visibility. Higher verbosity levels require more processing and are best enabled temporarily to investigate issues. Valid values are SHORT, SUMMARY, and DETAILED. Default: SHORT.
- queue_config: Prometheus maintains a sharded queue to batch series together for better performance and compression efficiency. These parameters control batching for processing and ingress efficiency by optimizing payload size and request frequency.
  - batch_send_deadline: Defines the period between requests to send items in the queue. Chronosphere recommends a value of 1s.
  - max_samples_per_send: Limits the number of samples that Prometheus sends in each request. Requests can be sent with fewer samples than this limit. Set to 2000 to force splitting requests that would include more than 2,000 items.

Configure multiple remote write endpoints

You can configure multiple remote write endpoints to help in a cutover from one target to another, or when migrating from v1 to v2. The following example demonstrates a configuration for a Prometheus Agent writing metrics to a remote write v1 endpoint and a remote write v2 endpoint.

remote_write:
  - url: https://...
    protobuf_message: prometheus.WriteRequest
    ...
 
  - url: https://<COMPANY>.chronosphere.io/data/api/v1/prom/remote/write
    protobuf_message: io.prometheus.write.v2.Request
    ...

Next steps

Verify that Observability Platform is receiving metrics.

If you encounter issues, review the Prometheus Agent logs and Troubleshooting.

Metrics and traces Configure collector