Discover and scrape Kubernetes resources
Configure the Collector using one of the following methods to monitor Kubernetes resources and ensure that your clusters are healthy.
If running in Kubernetes, you can configure the Collector to scrape kubelet or
cAdvisor metrics by setting the
true under the
kubeletMonitoring YAML collection.
port: Port the
kubeletis running on. Defaults to
bearerTokenFile: Path to file containing collector service account token. Defaults to
kubeletMetricsEnabled: Enables scraping kubelet metrics.
cadvisorMetricsEnabled: Enables scraping cAdvisor metrics.
probesMetricsEnabled: Enables collecting metrics on the status of liveness, readiness, and startup kubelet probes for Kubernetes containers.
labelsToAugment: Lists the metadata labels from pod labels the Collector adds to metrics.
annotationsToAugment: Lists the metadata labels from pod annotations the Collector adds to metrics.
By default, container-level metrics don't include metadata labels like
app, which searches can include when querying for these metrics. To automatically
add these labels from pod labels, use the
labelsToAugment flag to list the labels
the Collector adds to the metrics.
For example, to add the
app label to the container level metrics for a
DaemonSet deployment, use the following configuration under the
labelsToAugment: ["app", ...]
app="node-exporter" to these metrics, based on the following example
By default, container-level metrics don't include metadata labels, which searches can
include when querying for these metrics. To automatically add these labels from pod
annotations, use the
annotationsToAugment flag to list the labels the Collector
adds to the metrics.
For example, to add the
app_kubernetes_io_component label to the container-level
metrics for a
node-exporter DaemonSet deployment, use the following configuration
annotationsToAugment: ["app.kubernetes.io/component", ...]
app_kubernetes_io_component="infrastructure" to these metrics, assuming
the following example
Using Collector version 0.93.0 or later lets you specify pod labels and annotations you want to keep as a Prometheus label. This feature applies to pods only.
The following configuration example converts all pod labels called
my_label and all
pod annotations called
my.pod.annotation into Prometheus labels for the metrics
scraped from discovered pods. This is equivalent to a Prometheus
labelmap rule, but
sanitizes the label names and values:
- resource: "pod"
To discover endpoints in the
kube-system namespace, set the
kubeSystemEndpointsDiscoveryEnabled flag to
kube-system has many
constantly changing endpoints that may cause unnecessary load on the Collector, the
endpoint is disabled by default.
Use EndpointSlices (opens in a new tab) for Collector versions v0.85.0 or later. The Kubernetes cluster must use Kubernetes v1.19 or later to use EndpointSlices. Using EndpointSlices significantly reduces the amount of load on the Kubernetes API server.
If you modify a Collector manifest, you must update it in the cluster and restart the Collector.
You can use
ServiceMonitors to scrape
kube-state-metrics, which generate metrics
that track the health of deployments, nodes, and pods in a Kubernetes cluster.
Monitoring these metrics can help to ensure the health of your cluster because the
Collector expects to continually receive
kube-state-metrics. If the Collector can't
scrape these metrics, it's likely your Kubernetes cluster is experiencing issues you
need to resolve.
kube-state-metrics with a DaemonSet Collector is manageable for smaller
clusters, but can lead to out of memory (OOM) errors as the cluster scales.
Chronosphere recommends running the Collector as a sidecar to take advantage of
staleness markers. The following steps assume that:
- You're running a separate Collector as a Deployment to monitor
- You've already defined a Kubernetes Service and ServiceMonitor for
If you're already running the Collector as a DaemonSet, you must update the manifest for both Collector instances.
After installing the
ServiceMonitors CRD, complete the following steps to discover
Download this manifest (opens in a new tab).
datasection, replace the values for
api-tokenwith your Base64-encoded API token:
--- apiVersion: v1 data: address: <add-base64-encoded-token-here> api-token: <add-base64-encoded-token-here>
Apply the manifest:
kubectl apply -f path/to/ksm-chronocollector.yaml
Deploymentis started and running, and view the logs of the pod:
kubectl get pods NAME READY STATUS RESTARTS AGE chronocollector-jtgfw 1/1 Running 0 1m kubectl logs chronocollector-jtgfw ...
The Kubernetes API Server provides REST operations and a frontend to a cluster's shared state through which all other components interact. Unlike most other metrics emitted from a cluster, Kubernetes doesn't expose API Server metrics by using a pod, but instead exposes metrics directly from an endpoint in the API Server.
To ingest these metrics through traditional service discovery methods, you must
discover and scrape the endpoints directly. The Collector supports using
ServiceMonitors or job service discovery.
To discover and scrape API Server metrics using
enable both the
allowSkipPodInfo flag under the top level
serviceMonitor key and
endpointsDiscoveryEnabled flag under the
kubernetes in the
If using this method, deploy the Collector as an individual Kubernetes Deployment rather than a DaemonSet. This prevents multiple Collectors from scraping the same API Server metrics. Additionally, set the appropriate ServiceMonitor matching to prevent other Collectors from discovering the API Server ServiceMonitor.
To discover and scrape API Server metrics without using
ServiceMonitors, you can
jobs section of the Collector configuration for service discovery.
If using this method, deploy the Collector as an individual Kubernetes Deployment instead of as a DaemonSet. This prevents multiple Collectors from scraping the same API Server metrics.
The following is an example that discovers the API Server based on the value of the
__meta_kubernetes_pod_label_k8s_app label equal to
kube-apiserver (found in the
API Server Service object).
- name: kube-apiserver
- sourceLabels: [__name__]
- sourceLabels: [__meta_kubernetes_pod_node_name]