Files
In addition to the YAML configuration file that govern a pipeline’s settings, you can optionally add other kinds of files to pipelines, such as parsers, certificates, and custom Lua scripts.
Due to Kubernetes ConfigMap (opens in a new tab) storage limitations, the total size of a pipeline and its associated resources can’t exceed 1 MiB. This limit includes the combined size of configuration files, secrets, parsers, and any other files.
Add a file to a pipeline
To add a file to a pipeline, use one of these methods:
- Sign in to Telemetry Pipeline (opens in a new tab).
- Open the project that contains the pipeline that you want to add a file to.
- Go to Core Instances, then click the name of the Core Instance where you deployed the pipeline that you want to add a file to.
- Under Data Pipelines, click the name of the pipeline that you want to add a file to.
- Click Advanced Settings.
- Under Pipeline Files, click Upload File.
- Use one of these methods to upload a file:
- Click Select File to choose a file from your device.
- Enter a value in File name field and add content to the large text field to create a new file with that name and content.
- Optional: To encrypt your file, select the Encrypt file checkbox.
- Click Upload File.
Add a file to a Core Instance
Adding a file to a Core Instance does not circumvent the 1MiB resource limit for pipelines. If you add a file to a Core Instance, that file becomes a resource for every pipeline within that Core Instance, which increases the resource utilization for each pipeline accordingly.
You can also add a file to a Core Instance, which makes that file available to all pipeline within the Core Instance. To do so, use one of these methods:
- Sign in to Telemetry Pipeline (opens in a new tab).
- Open the project that contains the Core Instance that you want to add a file to.
- Go to Core Instances, then click the name of the Core Instance that you want to add a file to.
- Click Advanced Settings.
- Under Pipeline Files, click Upload File.
- Use one of these methods to upload a file:
- Click Select File to choose a file from your device.
- Enter a value in File name field and add content to the large text field to create a new file with that name and content.
- Optional: To encrypt your file, select the Encrypt file checkbox.
- Click Upload File.
Reference a file
After you’ve added a file to a pipeline, you can reference it in a pipeline’s configuration file by using the following variable:
{{ files.FILENAME }}Replace FILENAME with the name of your file.
File encryption
For encrypted files, Telemetry Pipeline uses RSA‑OAEP encryption with a 2048-bit key. Files are unencrypted by default, but you can choose to encrypt a file when you add to it a pipeline or Core Instance.