Secrets

When creating custom pipelines you might have settings that you don’t want to store in plain text or share with other users of the pipeline. Prevent this by using secrets in Chronosphere Telemetry Pipeline.

Due to Kubernetes ConfigMap storage limitations, the total size of a pipeline and its associated resources can’t exceed 1 MiB. This limit includes the combined size of configuration files, secrets, parsers, and any other files.

Secret storage

Telemetry Pipeline encrypts all stored secrets using RSA public key cryptography. For every Core Instance you create, Telemetry Pipeline generates a public key and a private key. Chronosphere stores the public key in the Telemetry Pipeline backend, and the private key is stored solely in your local environment. If you add a secret to a Core Instance or any of its pipelines, Chronosphere encrypts that secret with the Core Instance’s public key, then stores the encrypted version in the Telemetry Pipeline backend. When your Core Instance fetches information from the Telemetry Pipeline backend, it uses its private key to decrypt any encrypted secrets. Decrypted secrets are then stored within that Core Instance as Kubernetes Secrets, which take the form of plain text.

Chronosphere does not store private keys or unencrypted secrets in the Telemetry Pipeline backend. If you delete or modify the private key for a Core Instance, any attempts to decrypt secrets will fail due to incompatibility with the associated public key.As a protective measure, Chronosphere recommends backing up your private key in case you need to restore it later.

The following diagrams illustrate the secret generation and usage.

Encryption

Decryption

Add a secret

You can add a secret to an individual pipeline or to a Core Instance. Adding a secret to a Core Instance makes that secret available to all pipelines associated with that Core Instance.

Pipelines

Use one of the following methods to add a secret to a pipeline.

Web
Pipeline CLI

Sign in to Telemetry Pipeline.
Open the project that contains the pipeline that you want to add a secret to.
Go to Core Instances, then click the name of the Core Instance associated with the pipeline that you want to add a secret to.
Under Data Pipelines, click the name of the pipeline that you want to add a secret to.
Click Advanced Settings.
In the Add new secret section, enter values for the Name and Value fields.
Click Add secret to save the secret.

To add secrets to a pipeline in Pipeline CLI, use the following flags with the create pipeline or update pipeline command:

calyptia {create|update} pipeline \
--secrets-file FILE \
--secrets-format FORMAT

Replace the following:

FILE: The name of the file that contains your secrets.
FORMAT: The format of the file that contains your secrets. Possible values: auto, env, json, or yaml.

Core Instance

Use one of the following methods to add a secret to a Core Instance.

Web
Pipeline CLI
Helm

Sign in to Telemetry Pipeline.
Open the project that contains the Core Instance that you want to add a secret to.
Go to Core Instances, then click the name of the Core Instance that you want to add a secret to.
Click Advanced Settings.
In the Add new secret section, enter values the Name and Value fields.
Click Add secret to save the secret.

Any secrets you add are visible in the Older Secrets table.

To add a secret to a Core Instance in Pipeline CLI, use the following command:

calyptia create core_instance_secret \
--core-instance INSTANCE
--key SECRET_KEY
--value SECRET_VALUE

Replace the following:

INSTANCE: The name of your Core Instance.
SECRET_NAME: The key of your secret’s key/value pair.
SECRET_VALUE: The value of your secret’s key/value pair.

To add multiple secrets to a Core Instance, repeat this command for each key/value pair.

Core Instance secrets can be added using the coreSecret variable in Helm.

Update a secret

You can replace the value of an existing secret with an updated value.

Before you update a secret, you won’t be able to see its current value. Similarly, after you update a secret, you won’t be able to see any of its previous values.

Pipelines

Use one of the following methods to update a secret for a pipeline:

Web
Pipeline CLI

Sign in to Telemetry Pipeline.
Open the project that contains the pipeline whose secret you want to update.
Go to Core Instances, then click the name of the Core Instance associated with the pipeline whose secret you want to update.
Under Data Pipelines, click the name of the pipeline whose secret you want to update.
Click Advanced Settings.
In the Older Secrets table, find the secret that you want to update, then click Update.
Enter a new value in the Secret value field, then click Save changes.

You can update the secret by first listing all secrets, and then updating the secret with new values.To list all the secrets from a pipeline, use the command:

calyptia get pipeline_secrets --pipeline PIPELINE_ID --show-ids

The output is similar to the following:

ID                                   KEY            AGO
73a941d6-e658-4f12-a175-97d9063c466f es_http_passwd 5 minutes

Copy the secret ID and use it to update its value:

calyptia update pipeline_secret SECRET_ID NEW_VALUE

Core Instance

Use one of the following methods to update a secret for a Core Instance:

Web
Helm

Sign in to Telemetry Pipeline.
Open the project that contains the Core Instance whose secret you want to update.
Go to Core Instances, then click the name of the Core Instance whose secret you want to update.
Click Advanced Settings.
In the Older Secrets table, find the secret that you want to update, then click Update.
Enter a new value in the Secret value field, then click Save changes.

The secret’s name and date it was added display in the Older Secrets table.

Core Instance secrets can be updated using the coreSecret variable in Helm.

Reference a secret

In pipeline configuration files, you can reference both the values of Telemetry Pipeline-amanged secrets and Kubernetes-managed Secrets.

Telemetry Pipeline-managed secrets

If you add a secret to a pipeline or to a Core Instance, you can use {{ secrets.key }} syntax to reference the value of that secret in a configuration file. For example, given the following secret:

es-password=123456

You can reference the value of this secret in a configuration file without specifying this value in plain text. Instead, use the value {{ secrets.es-password }}:

pipeline:
  inputs:
    - Name: forward
      listen: 0.0.0.0
      port: "24284"
  outputs:
    - Name: es
      host: hostname.us-east-2.es.amazonaws.com
      port: "443"
      index: calyptia-core
      type: _doc
      http_user: es
      http_passwd: {{ secrets.es-password }}
      Match_Regex: .{0,}

From your pipeline’s perspective, the http_passwd key has a value of 123456.

Kubernetes-managed Secrets

If you deployed a pipeline in a Kubernetes cluster, you can reference the value of Kubernetes-managed Secrets from that cluster without adding any new secrets to Telemetry Pipeline directly. To reference the value of a Kubernetes Secret inside a pipeline configuration file, use the following syntax:

${SECRET_NAME_PARAMETER}

Replace the following:

NAME: The name of your Kubernetes Secret.
PARAMETER: The name of a key stored within the data field of your Kubernetes Secret.

For example, for the following Kubernetes Secret named test:

copy

apiVersion: v1
data:
  SPECIALSAUCE: ZHVtbXk=
kind: Secret
type: Opaque

You can reference the value of SPECIALSAUCE in a pipeline configuration file through ${SECRET_TEST_SPECIALSAUCE}:

pipeline:
    inputs:
        - dummy: {"message": "dummy"}
          rate: "1"
          samples: "0"
          start_time_sec: "-1"
          start_time_nsec: "-1"
          Name: ${SECRET_TEST_SPECIALSAUCE}

From your pipeline’s perspective, the Name key has a value of ZHVtbXk=.

Back up and restore a private key

To prevent issues with secret storage and decryption, you can back up the private keys stored within each Core Instance, then restore these keys later if needed.

Back up a private key

To back up the private key for a Core Instance:

Run the following kubectl command:
```
kubectl get secret "calyptia-INSTANCE-ENV-secret"  -o jsonpath='{.data.private-key}' | base64 --decode
```
Replace the following values:
- INSTANCE: The name of the Core Instance whose key you want to back up.
- ENV: The environment associated with your Core Instance. If you’re not sure what value to use here, your environment is likely default.
In the resulting output, look for the text that contains your private key:
```
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
```
Your private key is the value between BEGIN RSA PRIVATE KEY and END RSA PRIVATE KEY.
Copy the value of your private key and save it in a trusted, secure location, like a password manager or cloud secret storage service.

Restore a private key

To restore a private key that you previously backed up:

In kubectl, run the following command to encode your key:
```
ENCODED_PRIVATE_KEY=$(echo '-----BEGIN RSA PRIVATE KEY-----
VALUE
-----END RSA PRIVATE KEY-----' | base64)
```
Replace VALUE with the value of your private key.
Run the following command to delete any previously encoded secrets, which might be inaccurate due to an incorrect or deleted private key:
```
kubectl delete secret calyptia-INSTANCE-ENV-secret -n NAMESPACE
```
Replace the following values:
- INSTANCE: The name of the Core Instance whose key you want to restore.
- ENV: The environment associated with your Core Instance. If you’re not sure what value to use here, your environment is likely default.
- NAMESPACE: The namespace where you deployed your Core Instance. If you’re not sure what value to use here, your namespace is likely calyptia.

Run the following command to recreate your secrets using the restored private key:

echo "apiVersion: v1
kind: Secret
metadata:
labels:
   app.kubernetes.io/created-by: core-operator
   app.kubernetes.io/managed-by: core-operator
   app.kubernetes.io/part-of: calyptia
   calyptia.core: core-operator
name: calyptia-INSTANCE-ENV-secret
type: Opaque
data:
private-key: $ENCODED_PRIVATE_KEY" | kubectl apply -f -

Replace the following values:

INSTANCE: The name of the Core Instance whose key you want to restore.
ENV: The environment associated with your Core Instance. If you’re not sure what value to use here, your environment is likely default.

Observability Platform

Telemetry Pipeline

Tools

More information

Secret storage

Encryption

Decryption

Add a secret

Pipelines

Core Instance

Update a secret

Pipelines

Core Instance

Reference a secret

Telemetry Pipeline-managed secrets

Kubernetes-managed Secrets

Back up and restore a private key

Back up a private key

Restore a private key

Observability Platform

Telemetry Pipeline

Tools

More information

​Secret storage

​Encryption

​Decryption

​Add a secret

​Pipelines

​Core Instance

​Update a secret

​Pipelines

​Core Instance

​Reference a secret

​Telemetry Pipeline-managed secrets

​Kubernetes-managed Secrets

​Back up and restore a private key

​Back up a private key

​Restore a private key

Secret storage

Encryption

Decryption

Add a secret

Pipelines

Core Instance

Update a secret

Pipelines

Core Instance

Reference a secret

Telemetry Pipeline-managed secrets

Kubernetes-managed Secrets

Back up and restore a private key

Back up a private key

Restore a private key