TELEMETRY PIPELINE
Files

Files

In addition to the YAML configuration file that govern a pipeline's settings, you can optionally add other kinds of files to pipelines, such as parsers, certificates, and custom Lua scripts.

Due to Kubernetes ConfigMap (opens in a new tab) storage limitations, the total size of a pipeline and its associated resources can't exceed 1 MiB. This limit includes the combined size of configuration files, secrets, parsers, and any other files.

Add a file to a pipeline

To add a file to a pipeline, use one of these methods:

  1. Sign in to Telemetry Pipeline (opens in a new tab).
  2. Open the project that contains the pipeline that you want to add a file to.
  3. Go to Core Instances, then click the name of the Core Instance where you deployed the pipeline that you want to add a file to.
  4. Under Data Pipelines, click the name of the pipeline that you want to add a file to.
  5. Click  Advanced Settings.
  6. Under Pipeline Files, click Upload File.
  7. Use one of these methods to upload a file:
    • Click Select File to choose a file from your device.
    • Enter a value in File name field and add content to the large text field to create a new file with that name and content.
  8. Optional: To encrypt your file, select the Encrypt file checkbox.
  9. Click Upload File.

Add a file to a Core Instance

Adding a file to a Core Instance does not circumvent the 1MiB resource limit for pipelines. If you add a file to a Core Instance, that file becomes a resource for every pipeline within that Core Instance, which increases the resource utilization for each pipeline accordingly.

You can also add a file to a Core Instance, which makes that file available to all pipeline within the Core Instance. To do so, use one of these methods:

  1. Sign in to Telemetry Pipeline (opens in a new tab).
  2. Open the project that contains the Core Instance that you want to add a file to.
  3. Go to Core Instances, then click the name of the Core Instance that you want to add a file to.
  4. Click  Advanced Settings.
  5. Under Pipeline Files, click Upload File.
  6. Use one of these methods to upload a file:
    • Click Select File to choose a file from your device.
    • Enter a value in File name field and add content to the large text field to create a new file with that name and content.
  7. Optional: To encrypt your file, select the Encrypt file checkbox.
  8. Click Upload File.

Reference a file

After you've added a file to a pipeline, you can reference it in a pipeline's configuration file by using the following variable:

{{ files.FILENAME }}

Replace FILENAME with the name of your file.

File encryption

For encrypted files, Telemetry Pipeline uses RSA‑OAEP encryption with a 2048-bit key. Files are unencrypted by default, but you can choose to encrypt a file when you add to it a pipeline or Core Instance.