Accessing Secrets and Environment Variables

Accessing Secrets

Secrets can be attached to a pipeline or a function deployment as described in the DataEngine User Guide.

DataEngine makes secrets available to functions at runtime via ctx.secrets.

The ctx class can access secrets by their secret key. Inside your init(ctx) or handler(ctx, event), secrets are available as a nested dictionary on ctx.secrets. The top-level key matches the structure of secrets.yaml, the file that stores the secrets:

def init(ctx):
    # Access the top-level "secrets" key defined in secrets.yaml
    secrets_dict = ctx.secrets.get("secrets", {})

    # Read individual values from the secrets dictionary
    api_key = secrets_dict.get("MY_API_KEY", "")
    db_user = secrets_dict.get("DB_USERNAME", "")
    db_pass = secrets_dict.get("DB_PASSWORD", "")
    endpoint = secrets_dict.get("ENDPOINT_URL", "")

    # Validate required secrets before proceeding
    if not api_key:
        ctx.logger.error("MY_API_KEY is missing from secrets.yaml -- cannot start")
        raise RuntimeError("Missing required secret: MY_API_KEY")

    ctx.logger.info(f"API key loaded: {api_key[:4]}...{api_key[-4:]}")

Use .get() with a default of ““ so a missing key gives you an empty string instead of a KeyError. Then validate and fail fast with a clear message.

Logging Secrets Safely

To log secrets safely, avoid logging full secret values. Mask the secret values to show just enough for identification:

from openai import OpenAI
import boto3

def init(ctx):
    secrets = ctx.secrets

    # AI client using secret API key + env var for model selection
    ai_client = OpenAI(
        base_url=os.environ.get("LLM_ENDPOINT", ""),
        api_key=secrets.get("LLM_API_KEY", "")
    )

    # S3 client using secret credentials and endpoint
    s3_client = boto3.client(
        "s3",
        endpoint_url=secrets.get("VAST_OBJECT_ENDPOINT", ""),
        aws_access_key_id=secrets.get("VAST_ACCESS_KEY", ""),
        aws_secret_access_key=secrets.get("VAST_SECRET_KEY", "")
    )

Accessing Environment Variables

DataEngine users can configure or import environment variables to function deployments and to pipelines.

Environment variables are intended for non sensitive information that functions may need to access which may vary between deployments.

Your function code can read any environment variables that are imported for the deployment from 'os.environ'.

To access environment variables:

At the top of your function, import 'os', python’s built-in Operating System module, to give the function access to the system environment in which it is running.
```
import os
```
When you want to access a given environment variable, read it from 'os.environ'. For example:
```
sleep_time = int(os.environ.get("SLEEP_TIME", "0"))
```

Documentation Index

Accessing Secrets and Environment Variables