VAST DataEngine is the VAST compute orchestration framework. The framework enables users to write, deploy and manage execution pipelines against scheduled or event driven triggers. VAST DataEngine runs on VAST Cluster and dynamically provisions infrastructure and runtime resources according to demand. VAST DataEngine frees users from the need to manage and handle infrastructure scaling and availability or worry about performance.
VAST DataEngine features a web-based graphic user interface, a command line interface and a REST API service for configuring and managing triggers, functions and pipelines and for scheduling, running and monitoring pipelines.
VAST DataEngine features a telemetries service, providing observability over user executed pipelines and functions. The telemetries service collects logs, and traces of pipelines and enables users to query them.
VAST DataEngine can support limitless scope of use cases and a limitless scope of personas who might interact with the framework.
VAST DataEngine utilizes an event broker to stream events into functions. You can choose to use a third party event broker or the VAST Event Broker.
Deployment of VAST DataEngine involves connecting a VAST Cluster tenant to two external services: a container registry that stores images of functions to be deployed, and a Kubernetes cluster to consume the event broker events and execute the user functions with the above events.
VAST DataEngine supports functions written in Python3. Triggers can be set on any file name suffix and prefix and functions can be written to handle any type of file format. You can also build and deploy HTTP server images.