Secrets
When creating custom pipelines you might have settings that you don't want to store in plain text or share with other users of the pipeline. Prevent this by using secrets in Chronosphere Telemetry Pipeline.
Secret storage
Telemetry Pipeline encrypts all stored secrets using RSA public key cryptography.
Each deployment of Telemetry Pipeline generates a unique key pair (private and public) when a new instance registers. The private key is stored in your operating environment. When a pipeline is deployed or updated and requires a secret, that secret is fetched encrypted from Telemetry Pipeline and decrypted by the keys in your environment, using the in-memory private RSA key.
The Pipeline API doesn't store the private key in any form.
The following diagrams illustrate the secret generation and usage.
Encryption
Decryption
Add a secret
You can add a secret to an individual pipeline or to a Core Instance. Adding a secret to a Core Instance makes that secret available to all pipelines within the Core Instance.
Pipelines
Use one of the following methods to add a secret to a pipeline:
- Sign in to Telemetry Pipeline (opens in a new tab).
- Go to Core Instances, then click the name of the Core Instance where you deployed the pipeline that you want to add a secret to.
- Under Data Pipelines, click the name of the pipeline that you want to add a secret to.
- Click Advanced Settings.
- In the Add new secret section, enter values for the Name and Value fields.
- Click Add secret to save the secret.
Core Instance
Use one of the following methods to add a secret to a Core Instance:
- Sign in to Telemetry Pipeline (opens in a new tab).
- Go to Core Instances, then click the name of the Core Instance that you want to add a secret to.
- Click Advanced Settings.
- In the Add new secret section, enter values the Name and Value fields.
- Click Add secret to save the secret.
Any secrets you add are visible in the Older Secrets table.
Update a secret
You can replace the value of an existing secret with an updated value.
Before you update a secret, you won't be able to see its current value. Similarly, after you update a secret, you won't be able to see any of its previous values.
Pipelines
Use one of the following methods to update a secret for a pipeline:
- Sign in to Telemetry Pipeline (opens in a new tab).
- Go to Core Instances, then click the name of the Core Instance that contains the pipeline whose secret you want to update.
- Under Data Pipelines, click the name of the pipeline whose secret you want to update.
- Click Advanced Settings.
- In the Older Secrets table, find the secret that you want to update, then click Update.
- Enter a new value in the Secret value field, then click Save changes.
Core Instance
Use one of the following methods to update a secret for a Core Instance:
- Sign in to Telemetry Pipeline (opens in a new tab).
- Go to Core Instances, then click the name of the Core Instance that contains the secret you want to update.
- Click Advanced Settings.
- In the Older Secrets table, find the secret that you want to update, then click Update.
- Enter a new value in the Secret value field, then click Save changes.
The secret's name and date it was added display in the Older Secrets table.
Reference a secret
Secrets can be used as references by Pipeline CLI and Kubernetes.
Pipeline CLI
Secrets in Telemetry Pipeline are stored per pipeline and referenced in configuration paths by using brackets. For example, a secret that looks like the following:
key=value
can be referenced in a configuration as {{ secrets.key }}
. This syntax must
include a space between each set of brackets and the text inside them.
To reference secrets in Pipeline CLI, create a new file in your local environment and
then add it to your configuration. For example, in the pipeline-es.yaml
file, don't
specify HTTP_Passwd
in plain text. Instead, use a secret with the variable
es_http_passwd
.
pipeline:
inputs:
- Name: forward
listen: 0.0.0.0
port: "24284"
outputs:
- Name: es
host: hostname.us-east-2.es.amazonaws.com
port: "443"
index: calyptia-core
type: _doc
http_user: es
http_passwd: {{ secrets.es_http_passwd }}
Match_Regex: .{0,}
Define the es_http_passwd
variable in the secrets.env
file:
es_http_passwd=123123
Kubernetes Secrets
If you deployed Telemetry Pipeline in a Kubernetes cluster, you can reference existing Kubernetes Secrets (opens in a new tab) in that cluster without adding any new Secrets to Telemetry Pipeline directly.
To reference a Kubernetes Secret inside a pipeline configuration file, use the following syntax:
${SECRET_NAME_PARAMETER}
Replace the following:
NAME
: The name of your Kubernetes Secret.PARAMETER
: The name of a key stored within thedata
field of your Kubernetes Secret.
To prevent errors, be sure to remove any hyphens (-
) from the name of your
Kubernetes Secret before referencing it in Telemetry Pipeline, because NAME
can't
contain a hyphen.
For example, for the following Kubernetes Secret named test
:
apiVersion: v1
data:
SPECIALSAUCE: ZHVtbXk=
kind: Secret
metadata:
creationTimestamp: 2024-05-30T21:53:05Z
labels:
app.kubernetes.io/component: calyptia-core
core-pipeline: default.213-test-k8s-secrets
name: test
namespace: default
resourceVersion: "220374"
uid: 8b5f0c7e-0826-4c0b-90be-8bce36f026b3
type: Opaque
You can reference the value of SPECIALSAUCE
in a pipeline configuration file
through ${SECRET_TEST_SPECIALSAUCE}
:
pipeline:
inputs:
- dummy: {"message": "dummy"}
rate: "1"
samples: "0"
start_time_sec: "-1"
start_time_nsec: "-1"
Name: ${SECRET_TEST_SPECIALSAUCE}
From your pipeline's perspective, the Name
key has a value of ZHVtbXk=
.