TELEMETRY PIPELINE
Secrets

Secrets

When creating custom pipelines you might have settings that you don't want to store in plain text or share with other users of the pipeline. Prevent this by using secrets in Chronosphere Telemetry Pipeline.

Due to Kubernetes ConfigMap (opens in a new tab) storage limitations, the total size of a pipeline and its associated resources can't exceed 1 MiB. This limit includes the combined size of configuration files, secrets, parsers, and any other files.

Secret storage

Telemetry Pipeline encrypts all stored secrets using RSA public key cryptography.

For every Core Instance you create, Telemetry Pipeline generates a public key and a private key. Chronosphere stores the public key in the Telemetry Pipeline backend, and the private key is stored solely in your local environment.

If you add a secret to a Core Instance or any of its pipelines, Chronosphere encrypts that secret with the Core Instance's public key, then stores the encrypted version in the Telemetry Pipeline backend. When your Core Instance fetches information from the Telemetry Pipeline backend, it uses its private key to decrypt any encrypted secrets. Decrypted secrets are then stored within that Core Instance as Kubernetes Secrets (opens in a new tab), which take the form of plain text.

⚠️

Chronosphere does not store private keys or unencrypted secrets in the Telemetry Pipeline backend. If you delete or modify the private key for a Core Instance, any attempts to decrypt secrets will fail due to incompatibility with the associated public key.

As a protective measure, Chronosphere recommends backing up your private key in case you need to restore it later.

The following diagrams illustrate the secret generation and usage.

Encryption

Decryption

Add a secret

You can add a secret to an individual pipeline or to a Core Instance. Adding a secret to a Core Instance makes that secret available to all pipelines within the Core Instance.

Pipelines

Use one of the following methods to add a secret to a pipeline:

  1. Sign in to Telemetry Pipeline (opens in a new tab).
  2. Open the project that contains the pipeline that you want to add a secret to.
  3. Go to Core Instances, then click the name of the Core Instance where you deployed the pipeline that you want to add a secret to.
  4. Under Data Pipelines, click the name of the pipeline that you want to add a secret to.
  5. Click  Advanced Settings.
  6. In the Add new secret section, enter values for the Name and Value fields.
  7. Click Add secret to save the secret.

Core Instance

Use one of the following methods to add a secret to a Core Instance:

  1. Sign in to Telemetry Pipeline (opens in a new tab).
  2. Open the project that contains the Core Instance that you want to add a secret to.
  3. Go to Core Instances, then click the name of the Core Instance that you want to add a secret to.
  4. Click  Advanced Settings.
  5. In the Add new secret section, enter values the Name and Value fields.
  6. Click Add secret to save the secret.

Any secrets you add are visible in the Older Secrets table.

Update a secret

You can replace the value of an existing secret with an updated value.

Before you update a secret, you won't be able to see its current value. Similarly, after you update a secret, you won't be able to see any of its previous values.

Pipelines

Use one of the following methods to update a secret for a pipeline:

  1. Sign in to Telemetry Pipeline (opens in a new tab).
  2. Open the project that contains the pipeline whose secret you want to update.
  3. Go to Core Instances, then click the name of the Core Instance that contains the pipeline whose secret you want to update.
  4. Under Data Pipelines, click the name of the pipeline whose secret you want to update.
  5. Click  Advanced Settings.
  6. In the Older Secrets table, find the secret that you want to update, then click Update.
  7. Enter a new value in the Secret value field, then click Save changes.

Core Instance

Use one of the following methods to update a secret for a Core Instance:

  1. Sign in to Telemetry Pipeline (opens in a new tab).
  2. Open the project that contains the Core Instance whose secret you want to update.
  3. Go to Core Instances, then click the name of the Core Instance whose secret you want to update.
  4. Click  Advanced Settings.
  5. In the Older Secrets table, find the secret that you want to update, then click Update.
  6. Enter a new value in the Secret value field, then click Save changes.

The secret's name and date it was added display in the Older Secrets table.

Reference a secret

Secrets can be used as references by Pipeline CLI and Kubernetes.

Pipeline CLI

Secrets in Telemetry Pipeline are stored per pipeline and referenced in configuration paths by using brackets. For example, a secret that looks like the following:

key=value

can be referenced in a configuration as {{ secrets.key }}. This syntax must include a space between each set of brackets and the text inside them.

To reference secrets in Pipeline CLI, create a new file in your local environment and then add it to your configuration. For example, in the pipeline-es.yaml file, don't specify HTTP_Passwd in plain text. Instead, use a secret with the variable es_http_passwd.

pipeline-es.yaml
pipeline:
  inputs:
    - Name: forward
      listen: 0.0.0.0
      port: "24284"
  outputs:
    - Name: es
      host: hostname.us-east-2.es.amazonaws.com
      port: "443"
      index: calyptia-core
      type: _doc
      http_user: es
      http_passwd: {{ secrets.es_http_passwd }}
      Match_Regex: .{0,}

Define the es_http_passwd variable in the secrets.env file:

secrets.env
es_http_passwd=123123

Kubernetes Secrets

If you deployed Telemetry Pipeline in a Kubernetes cluster, you can reference existing Kubernetes Secrets (opens in a new tab) in that cluster without adding any new Secrets to Telemetry Pipeline directly.

To reference a Kubernetes Secret inside a pipeline configuration file, use the following syntax:

${SECRET_NAME_PARAMETER}

Replace the following:

  • NAME: The name of your Kubernetes Secret.
  • PARAMETER: The name of a key stored within the data field of your Kubernetes Secret.

To prevent errors, be sure to remove any hyphens (-) from the name of your Kubernetes Secret before referencing it in Telemetry Pipeline, because NAME can't contain a hyphen.

For example, for the following Kubernetes Secret named test:

test
apiVersion: v1
data:
  SPECIALSAUCE: ZHVtbXk=
kind: Secret
metadata:
  creationTimestamp: 2024-05-30T21:53:05Z
  labels:
    app.kubernetes.io/component: calyptia-core
    core-pipeline: default.213-test-k8s-secrets
  name: test
  namespace: default
  resourceVersion: "220374"
  uid: 8b5f0c7e-0826-4c0b-90be-8bce36f026b3
type: Opaque

You can reference the value of SPECIALSAUCE in a pipeline configuration file through ${SECRET_TEST_SPECIALSAUCE}:

special-pipeline.yaml
pipeline:
    inputs:
        - dummy: {"message": "dummy"}
          rate: "1"
          samples: "0"
          start_time_sec: "-1"
          start_time_nsec: "-1"
          Name: ${SECRET_TEST_SPECIALSAUCE}

From your pipeline's perspective, the Name key has a value of ZHVtbXk=.

Back up and restore a private key

To prevent issues with secret storage and decryption, you can back up the private keys stored within each Core Instance, then restore these keys later if needed.

Back up a private key

To back up the private key for a Core Instance:

  1. Run the following kubectl command:

    kubectl get secret "calyptia-INSTANCE-ENV-secret"  -o jsonpath='{.data.private-key}' | base64 --decode

    Replace the following values:

    • INSTANCE: The name of the Core Instance whose key you want to back up.
    • ENV: The environment associated with your Core Instance. If you're not sure what value to use here, your environment is likely default.
  2. In the resulting output, look for the text that contains your private key:

    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----

    Your private key is the value between BEGIN RSA PRIVATE KEY and END RSA PRIVATE KEY.

  3. Copy the value of your private key and save it in a trusted, secure location, like a password manager or cloud secret storage service.

If you have multiple Core Instances, repeat these steps for each Core Instance whose private key you want to back up.

Restore a private key

To restore a private key that you previously backed up:

  1. In kubectl, run the following command to encode your key:

    ENCODED_PRIVATE_KEY=$(echo '-----BEGIN RSA PRIVATE KEY-----
    VALUE
    -----END RSA PRIVATE KEY-----' | base64)

    Replace VALUE with the value of your private key.

  2. Run the following command to delete any previously encoded secrets, which might be inaccurate due to an incorrect or deleted private key:

    kubectl delete secret calyptia-INSTANCE-ENV-secret -n NAMESPACE

    Replace the following values:

    • INSTANCE: The name of the Core Instance whose key you want to restore.
    • ENV: The environment associated with your Core Instance. If you're not sure what value to use here, your environment is likely default.
    • NAMESPACE: The namespace where you deployed your Core Instance. If you're not sure what value to use here, your namespace is likely calyptia.
  3. Run the following command to recreate your secrets using the restored private key:

    echo "apiVersion: v1
    kind: Secret
    metadata:
    labels:
       app.kubernetes.io/created-by: core-operator
       app.kubernetes.io/managed-by: core-operator
       app.kubernetes.io/part-of: calyptia
       calyptia.core: core-operator
    name: calyptia-INSTANCE-ENV-secret
    type: Opaque
    data:
    private-key: $ENCODED_PRIVATE_KEY" | kubectl apply -f -

    Replace the following values:

    • INSTANCE: The name of the Core Instance whose key you want to restore.
    • ENV: The environment associated with your Core Instance. If you're not sure what value to use here, your environment is likely default.