VAST DataEngine is a serverless development framework that enables you to manage and deploy functions running against scheduled or event driven triggers, and to build functions and triggers into pipelines, schedule and deploy those pipelines. VAST DataEngine runs on VAST Cluster and dynamically provisions infrastructure and runtime resources according to demand. VAST DataEngine frees users from the need to manage and handle infrastructure scaling and availability or worry about performance.
VAST DataEngine features a web-based graphic user interface, a command line interface and a REST API service for configuring and managing triggers, functions and pipelines and for scheduling, running and monitoring pipelines.
VAST DataEngine features a telemetries service, providing observability over user executed pipelines and functions. The telemetries service collects logs, metrics and traces of pipelines and supports queries.
VAST DataEngine utilizes an event broker to stream triggered events into functions.
VAST DataEngine connects to external container registries, where images of your functions must be stored in order to be deployed, and to external Kubernetes clusters, which consume the event broker events and execute the user functions with the above events.
VAST DataEngine supports functions written in Python3. Triggers can be set on any file name suffix and prefix and functions can be written to handle any type of file format.
VAST DataEngine Resource Types
VAST DataEngine features the following types of resources:
Trigger. This is a resource that watches for a defined type of event and sends structured events, formatted as cloud events, to an event broker topic. VAST DataEngine enables the creation of two types of triggers:
Element triggers. These triggers watch for events where elements are added or removed from a view or where tags are added or removed from elements in a view.
Schedule triggers. These triggers issue events on a schedule.
Function. A function is a pointer to an image of a function stored on a container registry. VAST provides a CLI command to package a function that you can push to a container registry, ready to be referenced by a DataEngine function.
Function deployment. A function deployment is an instantiation of a function as part of a pipeline. Each deployment of any function within any pipeline can be configured independently.
Pipeline. A runtime instantiation of a set of functions and triggers that can be interconnected in a defined flow or in a group of defined flows.