TELEMETRY PIPELINE
Configure
Secrets

Secrets

When creating custom pipelines you might have settings that you don't want to store in plain text or share with other users of the pipeline. Prevent this by using secrets in Chronosphere Telemetry Pipeline.

Secret storage

Telemetry Pipeline encrypts all stored secrets using RSA public key cryptography.

Each deployment of Telemetry Pipeline generates a unique key pair (private and public) when a new instance registers. The private key is stored in your operating environment. When a pipeline is deployed or updated and requires a secret, that secret is fetched encrypted from Telemetry Pipeline and decrypted by the keys in your environment, using the in-memory private RSA key.

The Pipeline API doesn't store the private key in any form.

The following diagrams illustrate the secret generation and usage.

Encryption

Decryption

Create a secret

You can add a secret to Core Instances and Pipelines.

Core Instance

Use one of the following methods to create a secret for a Core Instance:

  1. Sign in to the Telemetry Pipeline web interface (opens in a new tab).
  2. Click Core Instances.
  3. In the Add new secret section, add a Name and Value.
  4. Click Add secret to save the secret.

The secret's name and date it was added display in the Older Secrets table.

Pipelines

Use one of the following methods to create a secret for a pipeline:

  1. Sign in to the Telemetry Pipeline web interface (opens in a new tab).

  2. Click Core Instances, and then select the Core Instance which has the Data Pipeline you want to add a secret to.

    If the pipeline doesn't exist, click Create a custom pipeline or Add new to create a new pipeline.

  3. Click the Pipeline Name of the pipeline to update. The pipeline summary appears.

  4. Click Advanced Settings.

  5. Add a Name and Value for your secret.

  6. Click Add secret.

Update a secret

You can replace the value of an existing secret with an updated value.

Secrets can be edited only with a new value, and previous values aren't displayed or stored.

Core Instance

Use one of the following methods to update a secret for a Core Instance:

  1. Sign in to the Telemetry Pipeline web interface (opens in a new tab).
  2. Click Core Instances.
  3. In the Add new secret section, add a Name and Value.
  4. Click Add secret to save the secret.

The secret's name and date it was added display in the Older Secrets table.

Pipelines

Use one of the following methods to update a secret for a pipeline:

  1. Sign in to the Telemetry Pipeline web interface (opens in a new tab).
  2. Click Core Instances, and then select the Core Instance that has the Data Pipeline you want to update.
  3. Click the Pipeline Name of the pipeline to update. The pipeline summary appears.
  4. Click Advanced Settings.
  5. Next to the secret, click Update.
  6. Add the Replacement Value.
  7. Click Save changes.

Reference a secret

Secrets can be used as references by Pipeline CLI and Kubernetes.

Pipeline CLI

Secrets in Telemetry Pipeline are stored per pipeline and referenced in configuration paths by using brackets. For example, a secret that looks like the following:

key=value

can be referenced in a configuration as {{ secrets.key }}. This syntax must include a space between each set of brackets and the text inside them.

To reference secrets in Pipeline CLI, create a new file in your local environment and then add it to your configuration. For example, in the pipeline-es.yaml file, don't specify HTTP_Passwd in plain text. Instead, use a secret with the variable es_http_passwd.

pipeline-es.yaml
pipeline:
  inputs:
    - Name: forward
      listen: 0.0.0.0
      port: "24284"
  outputs:
    - Name: es
      host: hostname.us-east-2.es.amazonaws.com
      port: "443"
      index: calyptia-core
      type: _doc
      http_user: es
      http_passwd: {{ secrets.es_http_passwd }}
      Match_Regex: .{0,}

Define the es_http_passwd variable in the secrets.env file:

secrets.env
es_http_passwd=123123

Kubernetes Secrets

If you deployed Telemetry Pipeline in a Kubernetes cluster, you can reference existing Kubernetes Secrets (opens in a new tab) in that cluster without adding any new Secrets to Telemetry Pipeline directly.

To reference a Kubernetes Secret inside a Telemetry Pipeline configuration file, use the following syntax:

${SECRET_NAME_PARAMETER}

Replace the following:

  • NAME: The name of your Kubernetes Secret.
  • PARAMETER: The name of a key stored within the data field of your Kubernetes Secret.

To prevent errors, be sure to remove any hyphens (-) from the name of your Kubernetes Secret before referencing it in Telemetry Pipeline, because NAME can't contain a hyphen.

For example, for the following Kubernetes Secret named test:

test
apiVersion: v1
data:
  SPECIALSAUCE: ZHVtbXk=
kind: Secret
metadata:
  creationTimestamp: 2024-05-30T21:53:05Z
  labels:
    app.kubernetes.io/component: calyptia-core
    core-pipeline: default.213-test-k8s-secrets
  name: test
  namespace: default
  resourceVersion: "220374"
  uid: 8b5f0c7e-0826-4c0b-90be-8bce36f026b3
type: Opaque

You can reference the value of SPECIALSAUCE in a pipeline configuration file through ${SECRET_TEST_SPECIALSAUCE}:

special-pipeline.yaml
pipeline:
    inputs:
        - dummy: {"message": "dummy"}
          rate: "1"
          samples: "0"
          start_time_sec: "-1"
          start_time_nsec: "-1"
          Name: ${SECRET_TEST_SPECIALSAUCE}

From your pipeline's perspective, the Name key has a value of ZHVtbXk=.

ScalingService types and ports