Secrets

When creating custom pipelines you may have settings that you don't want to store in plain text or share with other users of the pipeline. A good way to prevent this is by using secrets within Calyptia Core.

Secrets within Calyptia Core are stored per pipeline and referenced in configuration paths by using brackets. For example a secret that looks like the following:

key=value

Can be referenced in a configuration as {{ secrets.key }}.

Create a secret

Within the Pipeline Overview page, you can navigate to the Advanced Settings page to modify and add secrets.

To see this process in real time, follow along with an interactive demo (opens in a new tab).

Update a secret with a new value

You can update a secret by select update within the Advanced Settings page.

Secret storage

Calyptia Core stores all the secrets encrypted using RSA public key cryptography.

Each deployment of Calyptia Core has its unique key pair (private and public) generated when the new instance registers. The Calyptia Cloud API does not store this private key on any form.

When a new instance of Calyptia Core gets registered, a new unique key pair is generated and the private key is stored in your operating environment. When a pipeline is deployed or updated and it requires a secret, that secret is fetched encrypted from Calyptia Core and decrypted by the keys within your environment using the in-memory private RSA key.

The following diagrams illustrate the secret generation and usage.

Encryption

Decryption

Calyptia CLI

To reference secrets in Calyptia CLI, you must create a new file in your local environment and then add it within your configuration. For example, in the pipeline-es.conf file, you don't want to specify HTTP_Passwd in plain text. Instead, use a secret with the variable es_http_passwd. This variable is defined in the following secrets.env file:

[INPUT]
    Name          forward
    Host          0.0.0.0
    Port          24284

[OUTPUT]
    Name        es
    Match       *
    Host        hostname.us-east-2.es.amazonaws.com
    HTTP_User   es
    HTTP_Passwd {{secrets.es_http_passwd}}
    Port        443
    TLS         on

es_http_passwd=123123

Create a pipeline with a secret

When creating a pipeline, you must specify --secrets-file so that the pipeline configuration will be able to use them. The command for that is:

calyptia create pipeline --core-instance CORE_INSTANCE --config-file pipeline-es.conf --secrets-file secrets.env

Replace CORE_INSTANCE with the unique ID or name of the Calyptia Core instance.

Update a secret's value

If you need to update a secret, you can update the secret by first listing all secrets, and then updating the secret with new values.

List all the secrets from a pipeline:

calyptia get pipeline_secrets --pipeline PIPELINE_ID --show-ids

The output is similar to the following:

ID                                   KEY            AGO
73a941d6-e658-4f12-a175-97d9063c466f es_http_passwd 5 minutes

You can take the secret ID and use it to update its value:

calyptia update pipeline_secret SECRET_ID NEW_VALUE

Kubernetes Secrets

If you deployed Calyptia in a Kubernetes cluster, you can reference existing Kubernetes Secrets (opens in a new tab) in that cluster without adding any new Secrets to Calyptia directly.

To reference a Kubernetes Secret inside a Calyptia configuration file, use the following syntax:

${SECRET_NAME_PARAMETER}

Replace NAME with the name of your Kubernetes Secret and PARAMETER with the name of a key stored within the data field of your Kubernetes Secret.

For example, for the following Kubernetes Secret named test:

apiVersion: v1
data:
  specialSauce: ZHVtbXk=
kind: Secret
metadata:
  creationTimestamp: "2024-05-30T21:53:05Z"
  labels:
    app.kubernetes.io/component: calyptia-core
    core-pipeline: default.213-test-k8s-secrets
  name: test
  namespace: default
  resourceVersion: "220374"
  uid: 8b5f0c7e-0826-4c0b-90be-8bce36f026b3
type: Opaque

You can reference the value of specialSauce in a pipeline configuration file through ${SECRET_TEST_SPECIALSAUCE}:

pipeline:
    inputs:
        - dummy: {"message": "dummy" }
          rate: "1"
          samples: "0"
          start_time_sec: "-1"
          start_time_nsec: "-1"
          Name: ${SECRET_TEST_SPECIALSAUCE}

From your pipeline's perspective, the Name key has a value of ZHVtbXk=.