Working with Secrets and Environment Variables

VAST DataEngine supports secrets and environment variables, which are key-value pairs that can be accessed by DataEngine functions.

Secrets are stored encrypted and referenced by name. Environment variables are visible in the pipeline configuration.

Both secrets and environment variables can be made available to all functions in a pipeline or to specific function deployments within a pipeline.

The VAST DataEngine GUI accepts YAML file uploads to bulk-import environment variables and secrets into pipeline configurations or into specific function deployments. Alternatively, the GUI enables you to enter each key-value pair. If you are reusing common variables for multiple pipelines and functions, you will find the file upload option more practical.

Secrets

Use Secrets to Make Sensitive Values Available to Functions in Your Pipeline

VAST DataEngine functions support the use of a secrets file for storing API keys, credentials, access tokens and any other sensitive values that should not be hardcoded in source code or committed to version control. DataEngine can import this file at deployment time and make the contents available to functions at runtime via ctx.secrets.

Prepare a File for Bulk Importing Secrets

To bulk import secrets, prepare a file secrets.yaml containing the secrets as a flat map of key-value pairs under a "secrets: key":

secrets:
  "<key1>": "<value1>"
  "<key2>": "<value2>"
  ...
  "<keyn>": "<valuen>"

The following rules apply:

No further nesting of objects, lists, or YAML anchors. The GUI parser expects one level under secrets:.
Key names follow shell convention: uppercase, underscores, no spaces.
All keys and values must be quoted strings. This includes numeric and boolean values (`"1024"`, `"true"`).
The file must be valid YAML. You can run yamllint or python -c "import yaml; yaml.safe_load(open('import-env.yaml'))" to verify.

Example file content:

secrets:
  "MY_API_KEY": "sk-abc123..."
  "DB_USERNAME": "admin"
  "DB_PASSWORD": "hunter2"
  "ENDPOINT_URL": "http://10.0.0.1"

Access Secrets in Your Function

For details of how to access secrets in your function code, see Access to Secrets in the VAST DataEngine Runtime SDK Guide.

Environment Variables

Use Environment Variables for Non Sensitive Variable Configurations

VAST DataEngine functions support environment variables for non-sensitive configurations that you may want to change between deployments, such as model names, feature flags, connection endpoints, bucket names, schema names, and tuning parameters. Certain environment variables are required by all functions that connect to a VAST Database. Others are commonly needed. You can add variables with any custom keys your function expects — there is no fixed schema for optional variables.

Environment Variables for Database Functions

Functions that connect to a VAST Database may need the following environment variables:

Key	Description	Example
VAST_ENDPOINT	Query-engine URL (include protocol and port if non-default)	"http://queryengine.mycluster.vastdata.com"
VAST_ACCESS_KEY	S3-compatible access key for authentication (alternatively, this and VAST_SECRET_KEY can be secrets instead of environment variables)	"ABCD1234EFGH5678"
VAST_SECRET_KEY	Secret key corresponding to the S3-compatible secret key (alternatively, this and VAST_ACCESS_KEY can be secrets instead of environment variables)	"s3cr3t/k3y+value"
VAST_BUCKET	Target bucket (must already exist on the cluster)	"vast-business-engine"
VAST_SCHEMA	Database schema within the bucket	"interactions"

Functions often require the following environment variables. Add them as needed. They are not parsed by the DataEngine itself — your function code reads them from `os.environ`.

Key	Description	Example
VECTOR_DIMENSIONS	Embedding vector width	"1024"
SCOPE_HOURS	Lookback window for fetch functions	"1"
SCOPE_MINUTES	Finer-grained lookback window	"90"
WRITE_BATCH_SIZE	Rows per write transaction	"50"
MAX_WORKERS	Concurrency limit for parallel processing	"5"

Prepare a File for Bulk Importing Environment Variables

To bulk import environment variables, prepare a file import-env.yaml containing the variables as a flat map of key-value pairs.