Secrets
When creating custom pipelines you might have settings that you don't want to store in plain text or share with other users of the pipeline. A good way to prevent this is by using secrets within Chronosphere Telemetry Pipeline.
Secrets in Telemetry Pipeline are stored per pipeline and referenced in configuration paths by using brackets. For example a secret that looks like the following:
key=value
Can be referenced in a configuration as {{ secrets.key }}.
Create a secret
Within the Pipeline Overview page, you can navigate to the Advanced Settings page to modify and add secrets.
To see this process in real time, follow along with an interactive demo (opens in a new tab).
Update a secret with a new value
You can update a secret by select update within the Advanced Settings page.
Secrets can be edited only with a new value, and previous values are neither displayed nor stored.
Secret storage
Telemetry Pipeline stores all the secrets encrypted using RSA public key cryptography.
Each deployment of Telemetry Pipeline has its unique key pair (private and public) generated when the new instance registers. The Calyptia API doesn't store this private key on any form.
When a new instance of Telemetry Pipeline gets registered, a new unique key pair is generated and the private key is stored in your operating environment. When a pipeline is deployed or updated and it requires a secret, that secret is fetched encrypted from Telemetry Pipeline and decrypted by the keys within your environment using the in-memory private RSA key.
The following diagrams illustrate the secret generation and usage.
Encryption
Decryption
Calyptia CLI
To reference secrets in Calyptia CLI, create a new file in your local environment and
then add it to your configuration. For example, in the pipeline-es.yaml
file, don't
specify HTTP_Passwd
in plain text. Instead, use a secret with the variable
es_http_passwd
. This variable is defined in the following secrets.env
file:
pipeline:
inputs:
- Name: forward
listen: 0.0.0.0
port: "24284"
outputs:
- Name: es
host: hostname.us-east-2.es.amazonaws.com
port: "443"
index: calyptia-core
type: _doc
http_user: es
http_passwd: {{secrets.es_http_passwd}}
Match_Regex: .{0,}
es_http_passwd=123123
Create a pipeline with a secret
When creating a pipeline, you must specify --secrets-file
so that the pipeline
configuration can use them. Use the following command format:
calyptia create pipeline --core-instance CORE_INSTANCE --config-file pipeline-es.yaml --secrets-file secrets.env
Replace CORE_INSTANCE
with the unique ID or name of the Telemetry Pipeline
instance.
Update a secret's value
If you need to update a secret, you can update the secret by first listing all secrets, and then updating the secret with new values.
To list all the secrets from a pipeline, use this command:
calyptia get pipeline_secrets --pipeline PIPELINE_ID --show-ids
The output is similar to the following:
ID KEY AGO
73a941d6-e658-4f12-a175-97d9063c466f es_http_passwd 5 minutes
You can take the secret ID and use it to update its value:
calyptia update pipeline_secret SECRET_ID NEW_VALUE
Kubernetes Secrets
If you deployed Telemetry Pipeline in a Kubernetes cluster, you can reference existing Kubernetes Secrets (opens in a new tab) in that cluster without adding any new Secrets to Telemetry Pipeline directly.
To reference a Kubernetes Secret inside a Telemetry Pipeline configuration file, use the following syntax:
${SECRET_NAME_PARAMETER}
Replace the following:
NAME
: The name of your Kubernetes Secret.PARAMETER
: The name of a key stored within thedata
field of your Kubernetes Secret.
To prevent errors, be sure to remove any hyphens (-
) from the name of your
Kubernetes Secret before referencing it in Telemetry Pipeline, because NAME
can't
contain a hyphen.
For example, for the following Kubernetes Secret named test
:
apiVersion: v1
data:
specialSauce: ZHVtbXk=
kind: Secret
metadata:
creationTimestamp: "2024-05-30T21:53:05Z"
labels:
app.kubernetes.io/component: calyptia-core
core-pipeline: default.213-test-k8s-secrets
name: test
namespace: default
resourceVersion: "220374"
uid: 8b5f0c7e-0826-4c0b-90be-8bce36f026b3
type: Opaque
You can reference the value of specialSauce
in a pipeline configuration file
through ${SECRET_TEST_SPECIALSAUCE}
:
pipeline:
inputs:
- dummy: {"message": "dummy" }
rate: "1"
samples: "0"
start_time_sec: "-1"
start_time_nsec: "-1"
Name: ${SECRET_TEST_SPECIALSAUCE}
From your pipeline's perspective, the Name
key has a value of ZHVtbXk=
.