Implementing a VAST DataEngine Pipeline

VAST DataEngine enables you to assemble pipelines to implement your business logic through an interplay of runtime features, function code and invocation flows.

Implementing pipeline capabilities with VAST DataEngine is a shared process split between your function code and the platform interface. Awareness of both the platform capabilities and the developer ergonomics will help you decide how you want to construct your pipeline and save you time authoring your code.

DataEngine handles orchestration, routing, and visualization through platform features that rely on what your code explicitly exposes—such as structured return flags, custom metrics, and formatted logs. The runtime SDK provides built-in conveniences that help you author your code to support and enable the platform features.   

You will want to author your function code with awareness of downstream platform capabilities as well as the runtime SDK's built-in conveniences. These cross-domain features save you work in the authoring of the code as well as the building of the end to end data pipeline.

A DataEngine pipeline watches for and consumes events, and then routes and processes those events through a flow of functions that you design according to your business logic.

A configurable resource called a trigger watches for the initial events. Each consumed event is wrapped in a `VastEvent` object, formatted according to the CloudEvents v1.0 specification standard. The object format provides convenient access to event properties.  

Functions at all stages of a pipeline receive events in this format. Functions are invoked according to how you route them in the pipeline. Any given function in the pipeline may be invoked directly by a trigger, receiving the initial events, or it may be invoked by another function that has already received triggered events, processed them, and returned them for further processing.

Implementation of basic event processing is split across domains as follows:

In order to capture events for processing through your pipeline, you use triggers. A trigger is a resource that watches for a defined type of event and sends structured events, formatted as cloud events, to an event broker topic. VAST DataEngine enables you to create two types of triggers:

The CloudEvent structure has generic attributes and attributes that vary depending on the type of trigger or the function from which the event is received. You can access these attributes from your function code.

Each trigger type inserts some specific attributes into the event structure.

Implementation of triggers is split across domains as follows:

A CloudEvent with standard attributes:

Note

In raw CloudEvent JSON payloads, the partition key extension attribute is named "partitionkey" (without underscores, per CloudEvent specification conventions). When accessing this field in Python code using the DataEngine SDK, use the property event.partition_key.

{
  "specversion": "1.0",
  "id": "unique-event-id",
  "source": "vastdata.com:TriggerType.trigger-id",
  "type": "vastdata.com:TriggerType.SubType",
  "subject": "broker-name.topic-name",
  "time": "2026-03-22T10:30:00.123Z",
  "datacontenttype": "application/json",
  "partitionkey": "partition-value",
  "data": {
    "message": "event payload"
  }
}

A CloudEvent produced by an element trigger:

{
  "specversion": "1.0",
  "id": "abc123def456",
  "source": "vastdata.com:Element.my-trigger-id",
  "type": "vastdata.com:Element.ObjectCreated",
  "subject": "kafka-broker.my-topic",
  "time": "2026-03-22T10:30:00.123Z",
  "datacontenttype": "application/json",
  "elementpath": "my-bucket/path/to/object.txt",
  "partitionkey": "my-bucket/path/to/object.txt",
  "data": {
    "bucket": "my-bucket",
    "key": "path/to/object.txt"
  }
}

A CloudEvent produced by a schedule trigger:

{
  "specversion": "1.0",
  "id": "def456ghi789",
  "source": "vastdata.com:Schedule.my-schedule-trigger",
  "type": "vastdata.com:Schedule.TimerElapsed",
  "subject": "kafka-broker.schedule-topic",
  "time": "2026-03-22T10:30:00.123Z",
  "datacontenttype": "application/json",
  "cronschedule": "0 * * * *",
  "timerelapsedtimestamp": "2026-03-22T10:00:00.000Z",
  "partitionkey": "schedule-key",
  "data": {
    "message": "Scheduled execution"
  }
}

A CloudEvent returned by a function:

{
  "specversion": "1.0",
  "id": "ghi789jkl012",
  "source": "vastdata.com:upstream-function-name",
  "type": "vastdata.com:Function",
  "subject": "kafka-broker.function-topic",
  "time": "2026-03-22T10:30:00.123Z",
  "datacontenttype": "application/json",
  "partitionkey": "partition-value",
  "data": {
    "result": "output from upstream function",
    "processed_count": 42
  }
}

The runtime provides an embedded OpenTelemetry logger that you can use in your function code to log events attributes and define their severity.Logs and Traces

Logs generated by the embedded logger in deployed functions can be accessed by the DataEngine interfaces. Specifically, they are displayed in the VAST DataEngine GUI.

The runtime provides an embedded OpenTelemetry tracer that you can use in your function code to generate telemetry traces. In your function code, you can explicitly wrap key operations that you want to monitor, and you can tag spans with metadata.Logs and Traces

Traces generated by the embedded tracer in your functions can be visualized by the DataEngine API.

The DataEngine GUI visualizes the traces as waterfall graphs and flame charts, and enables you to inspect nested spans. It also lets you inspect traces associated with specific logs and vice versa.

VAST DataEngine supports storing key-value pairs for functions to access at runtime. These are supported as secrets for sensitive credentials and environment variables for non sensitive configurations. Secrets are stored with encryption and made available at runtime. They can be entered or uploaded when you configure the pipeline or per function deployment when you build the pipeline. Environment variables can be stored in the configuration of the pipeline or per function deployment when you build a pipeline.

Both secrets and environment variables can be made available to all functions in a pipeline or to specific function deployments within a pipeline.

The VAST DataEngine GUI accepts YAML file uploads to bulk-import environment variables and secrets into pipeline configurations or into specific function deployments. Alternatively, the GUI enables you to enter each key-value pair. If you are reusing common variables for multiple pipelines and functions, you will find the file upload option more practical.

implementation is split across domains as follows:

Batch event processing allows your DataEngine functions to aggregate incoming events and process them in a single execution instance. Batch event processing can help you control the pace of invocations and reduce the resources required to process events. This pattern is ideal for reducing high-overhead operations—such as bulk database writes, vector database indexing, or external multi-payload HTTP requests (e.g., LLM batch inference).

Batch event processing is configurable for each function deployment in each pipeline. When it is enabled for a given function deployment, incoming events are accumulated based on an event count threshold or a time window, whichever occurs first. The runtime invokes the function, passing a batch object to the function containing all the events that were accumulated before one of the thresholds was reached.

The function must be coded to return a list of processed events, which DataEngine dispatches downstream individually. Subsequent functions in the pipeline can buffer and re-batch events according to their own batch settings.

Traces of event batches are visualized in the traces page of the DataEngine UI, as well as the individual events in the batch. Batches have their own trace IDs.

Implementation of batch event processing is split across domains as follows:

Data pipelines can require differentiated flows based on various event related criteria and complex logic. The VAST DataEngine runtime supports this complexity with a conditional function routing feature, where you can use event trigger labels to route events conditionally from function to function.

You do this by setting trigger labels on events in a function's code and applying labels to the connectors that route events to other functions in the pipeline for further processing.

The custom metrics feature lets you emit custom metrics, retrieve them and visualize them. The runtime SDK wraps an OpenTelemetry metrics handler for four type of custom metrics.