> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chronosphere.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Secrets
When creating custom pipelines you might have settings that you don't want to store
in plain text or share with other users of the pipeline. Prevent this by using
secrets in Chronosphere Telemetry Pipeline.
Due to Kubernetes [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/)
storage limitations, the total size of a pipeline and its associated resources
can't exceed 1 MiB. This limit includes the combined size of
[configuration files](/ingest/pipeline/v2/configure/config-files), secrets, [parsers](/ingest/pipeline/v2/build/parsers),
and [any other files](/ingest/pipeline/v2/configure/files).
## Secret storage
Telemetry Pipeline encrypts all stored secrets using RSA public key cryptography.
For every [Core Instance](/ingest/pipeline/v2/concepts#core-operators-and-core-instances)
you create, Telemetry Pipeline generates a public key and a private key. Chronosphere
stores the public key in the Telemetry Pipeline backend, and the private key
is stored solely in your local environment.
If you add a secret to a Core Instance or any of its pipelines, Chronosphere
encrypts that secret with the Core Instance's public key, then stores the
encrypted version in the Telemetry Pipeline backend. When your Core Instance
fetches information from the Telemetry Pipeline backend, it uses its private key
to decrypt any encrypted secrets. Decrypted secrets are then stored within that
Core Instance as Kubernetes [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/),
which take the form of plain text.
Chronosphere doesn't store private keys or unencrypted secrets in the Telemetry
Pipeline backend. If you delete or modify the private key for a Core Instance,
any attempts to decrypt secrets will fail due to incompatibility with the associated
public key.
As a protective measure, Chronosphere recommends
[backing up your private key](#back-up-and-restore-a-private-key) in case you
need to restore it later.
The following diagrams illustrate the secret generation and usage.
### Encryption
```mermaid actions={false} theme={null}
graph TB
accTitle: How Telemetry Pipeline encrypts secret keys
accDescr: After a Telemetry Pipeline user creates or updates a secret, Telemetry Pipeline encrypts that secret using a public key, then stores the encrypted secret.
A[User]--Creates or updates secret-->B((Telemetry Pipeline));
B-.Uses public key to encrypt secret.->C[Aggregator public key]
C-.Stores encrypted secret.->B
style A stroke:#495057,stroke-width:2px
style B stroke:#495057,stroke-width:2px
style C stroke:#495057,stroke-width:2px
```
### Decryption
```mermaid actions={false} theme={null}
graph TB
accTitle: How Core Instances decrypt secret keys
accDescr: Telemetry Pipeline sends the encrypted pipeline secret to a Core Instance, and then that Core Instance decrypts that secret using a private key.
A((Telemetry Pipeline))-.Sends encrypted pipeline secret.->B[Core Instance]
B--Fetches pipeline secret-->A
B--Uses private key to decrypt secret-->C[Aggregator private key]
C-.Sends decrypted secret.->B
style A stroke:#495057,stroke-width:2px
style B stroke:#495057,stroke-width:2px
style C stroke:#495057,stroke-width:2px
```
## Add a secret
You can add a secret to an individual pipeline or to a [Core Instance](/ingest/pipeline/v2/concepts#core-operators-and-core-instances).
Adding a secret to a Core Instance makes that secret available to all pipelines
associated with that Core Instance.
Pipelines
Use one of the following methods to add a secret to a pipeline.
1. Sign in to [Telemetry Pipeline](https://core.calyptia.com/).
2. Open the project that contains the pipeline that you want to add a secret to.
3. Go to **Core Instances**, then click the name of the Core Instance associated
with the pipeline that you want to add a secret to.
4. Under **Data Pipelines**, click the name of the pipeline that you want to add
a secret to.
5. Click **Advanced Settings**.
6. In the **Add new secret** section, enter values for the **Name** and **Value**
fields.
7. Click **Add secret** to save the secret.
To add secrets to a pipeline in [Pipeline CLI](/ingest/pipeline/pipeline-cli),
use the following flags with the `create pipeline` or `update pipeline` command:
```shell /FILE/ /FORMAT/ theme={null}
calyptia {create|update} pipeline \
--secrets-file FILE \
--secrets-format FORMAT
```
Replace the following:
* *`FILE`*: The name of the file that contains your secrets.
* *`FORMAT`*: The format of the file that contains your secrets. Possible values:
`auto`, `env`, `json`, or `yaml`.
Core Instance
Use one of the following methods to add a secret to a Core Instance.
1. Sign in to [Telemetry Pipeline](https://core.calyptia.com/).
2. Open the project that contains the Core Instance that you want to add a secret to.
3. Go to **Core Instances**, then click the name of the Core Instance that you
want to add a secret to.
4. Click **Advanced Settings**.
5. In the **Add new secret** section, enter values for the **Name** and **Value**
fields.
6. Click **Add secret** to save the secret.
Any secrets you add are visible in the **Older Secrets** table.
To add a secret to a Core Instance in [Pipeline CLI](/ingest/pipeline/pipeline-cli),
use the following command:
```shell /INSTANCE/ /SECRET_KEY/ /SECRET_VALUE/ theme={null}
calyptia create core_instance_secret \
--core-instance INSTANCE \
--key SECRET_KEY \
--value SECRET_VALUE
```
Replace the following:
* *`INSTANCE`*: The name of your Core Instance.
* *`SECRET_KEY`*: The key of your secret's key/value pair.
* *`SECRET_VALUE`*: The value of your secret's key/value pair.
To add multiple secrets to a Core Instance, repeat this command for each key/value pair.
Core Instance secrets can be added using the `coreSecret` variable in
[Helm](/ingest/pipeline/v2/configure/helm).
## Update a secret
You can replace the value of an existing secret with an updated value.
Before you update a secret, you won't be able to see its current value. Similarly,
after you update a secret, you won't be able to see any of its previous values.
Pipelines
Use one of the following methods to update a secret for a pipeline:
1. Sign in to [Telemetry Pipeline](https://core.calyptia.com/).
2. Open the project that contains the pipeline whose secret you want to update.
3. Go to **Core Instances**, then click the name of the Core Instance associated
with the pipeline whose secret you want to update.
4. Under **Data Pipelines**, click the name of the pipeline whose secret you want
to update.
5. Click **Advanced Settings**.
6. In the **Older Secrets** table, find the secret that you want to update,
then click **Update**.
7. Enter a new value in the **Secret value** field, then click **Save changes.**
You can update the secret by first listing all secrets, and then updating the secret
with new values.
To list all the secrets from a pipeline, use the command:
```shell theme={null}
calyptia get pipeline_secrets --pipeline PIPELINE_ID --show-ids
```
The output is similar to the following:
```text theme={null}
ID KEY AGO
73a941d6-e658-4f12-a175-97d9063c466f es_http_passwd 5 minutes
```
Copy the secret `ID` and use it to update its value:
```shell theme={null}
calyptia update pipeline_secret SECRET_ID NEW_VALUE
```
Core Instance
Use one of the following methods to update a secret for a Core Instance:
1. Sign in to [Telemetry Pipeline](https://core.calyptia.com/).
2. Open the project that contains the Core Instance whose secret you want to update.
3. Go to **Core Instances**, then click the name of the Core Instance whose
secret you want to update.
4. Click **Advanced Settings**.
5. In the **Older Secrets** table, find the secret that you want to update,
then click **Update**.
6. Enter a new value in the **Secret value** field, then click **Save changes.**
The secret's name and date it was added display in the **Older Secrets** table.
Core Instance secrets can be updated using the `coreSecret` variable in
[Helm](/ingest/pipeline/v2/configure/helm).
## Reference a secret
In [pipeline configuration files](/ingest/pipeline/v2/configure/config-files), you can reference
both the values of Telemetry Pipeline-managed secrets and Kubernetes-managed Secrets.
### Telemetry Pipeline-managed secrets
If you [add a secret](#add-a-secret) to a pipeline or to a Core Instance, you can
use `{{ secrets.key }}` syntax to reference the value of that secret in a configuration
file.
For example, given the following secret:
```text filename="secrets.env" theme={null}
es-password=123456
```
You can reference the value of this secret in a configuration file without specifying
this value in plain text. Instead, use the value `{{ secrets.es-password }}`:
```yaml filename="es-pipeline.yaml" theme={null}
pipeline:
inputs:
- Name: forward
listen: 0.0.0.0
port: "24284"
outputs:
- Name: es
host: hostname.us-east-2.es.amazonaws.com
port: "443"
index: calyptia-core
type: _doc
http_user: es
http_passwd: {{ secrets.es-password }}
Match_Regex: .{0,}
```
From your pipeline's perspective, the `http_passwd` key has a value of `123456`.
### Kubernetes-managed Secrets
If you deployed a pipeline in a Kubernetes cluster, you can reference
the value of Kubernetes-managed [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
from that cluster without adding any new secrets to Telemetry Pipeline directly.
To reference the value of a Kubernetes Secret inside a [pipeline configuration file](/ingest/pipeline/v2/configure/config-files),
use the following syntax:
```yaml theme={null}
${SECRET_NAME_PARAMETER}
```
Replace the following:
* *`NAME`*: The name of your Kubernetes Secret.
* *`PARAMETER`*: The name of a key stored within the `data` field of your Kubernetes
Secret.
For example, for the following Kubernetes Secret named `test`:
```yaml filename="test" copy theme={null}
apiVersion: v1
data:
SPECIALSAUCE: ZHVtbXk=
kind: Secret
type: Opaque
```
You can reference the value of `SPECIALSAUCE` in a pipeline configuration file
through `${SECRET_TEST_SPECIALSAUCE}`:
```yaml filename="special-pipeline.yaml" theme={null}
pipeline:
inputs:
- dummy: {"message": "dummy"}
rate: "1"
samples: "0"
start_time_sec: "-1"
start_time_nsec: "-1"
Name: ${SECRET_TEST_SPECIALSAUCE}
```
From your pipeline's perspective, the `Name` key has a value of `ZHVtbXk=`.
## Back up and restore a private key
To prevent issues with [secret storage and decryption](#secret-storage), you can
back up the private keys stored within each Core Instance, then restore these
keys later if needed.
### Back up a private key
To back up the private key for a Core Instance:
1. Run the following kubectl command:
```shell /INSTANCE/ /ENV/ theme={null}
kubectl get secret "calyptia-INSTANCE-ENV-secret" -o jsonpath='{.data.private-key}' | base64 --decode
```
Replace the following values:
* *`INSTANCE`*: The name of the Core Instance whose key you want to back up.
* *`ENV`*: The environment associated with your Core Instance. If you're not
sure what value to use here, your environment is likely `default`.
2. In the resulting output, look for the text that contains your private key:
```text theme={null}
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
```
Your private key is the value between `BEGIN RSA PRIVATE KEY` and `END RSA PRIVATE KEY`.
3. Copy the value of your private key and save it in a trusted, secure location,
like a password manager or cloud secret storage service.
### Restore a private key
To restore a private key that you previously backed up:
1. In kubectl, run the following command to encode your key:
```shell /VALUE/ theme={null}
ENCODED_PRIVATE_KEY=$(echo '-----BEGIN RSA PRIVATE KEY-----
VALUE
-----END RSA PRIVATE KEY-----' | base64)
```
Replace *`VALUE`* with the value of your private key.
2. Run the following command to delete any previously encoded secrets, which might
be inaccurate due to an incorrect or deleted private key:
```shell /INSTANCE/ /ENV/ /NAMESPACE/ theme={null}
kubectl delete secret calyptia-INSTANCE-ENV-secret -n NAMESPACE
```
Replace the following values:
* *`INSTANCE`*: The name of the Core Instance whose key you want to restore.
* *`ENV`*: The environment associated with your Core Instance. If you're not
sure what value to use here, your environment is likely `default`.
* *`NAMESPACE`*: The [namespace](/ingest/pipeline/v2/configure/kubernetes/namespaces)
where you deployed your Core Instance. If you're not sure what value to use
here, your namespace is likely `calyptia`.
3. Run the following command to recreate your secrets using the restored private key:
```shell /INSTANCE/ /ENV/ theme={null}
echo "apiVersion: v1
kind: Secret
metadata:
labels:
app.kubernetes.io/created-by: core-operator
app.kubernetes.io/managed-by: core-operator
app.kubernetes.io/part-of: calyptia
calyptia.core: core-operator
name: calyptia-INSTANCE-ENV-secret
type: Opaque
data:
private-key: $ENCODED_PRIVATE_KEY" | kubectl apply -f -
```
Replace the following values:
* *`INSTANCE`*: The name of the Core Instance whose key you want to restore.
* *`ENV`*: The environment associated with your Core Instance. If you're not
sure what value to use here, your environment is likely `default`.