Building and Deploying a Pipeline on VAST DataEngine

Create the Pipeline Resource

From the left navigation menu, select Pipeline Management.
Click Create New Pipeline

Complete the fields:

Pipeline Name	Enter a name for the pipeline.
Description	Enter a description for the pipeline.
Kubernetes Cluster	From the dropdown, select the Kubernetes cluster that you want to use to execute the pipeline.
Namespace	Select which namespace to use.
Pipeline Secret Keys	Use this field to pass a secret to all function deployments in the pipeline. The secret can contain multiple key-value pairs and it is intended for sensitive information such as access keys for accessing data and metadata of S3 buckets. For details of how to work with secrets and environment variables, see Working with Secrets and Environment Variables. Note You can also specify secrets for individual function deployments in their individual configurations. You can enable functions to access these secret keys through use of the VAST DataEngine runtime SDK ctx.secrets property in your function code. For details about how to do this in your code, see the VAST DataEngine Runtime SDK Guide. To add secrets, do one of the following: Enter the secrets: Click Create secret. Enter the secret name in the Secret Name field. In the Key and Value fields provided, enter a key and its value. Click the add button provided to add another key-value pair row as needed until you have added all keys needed for the secret. To add another secret, click Create Another secret and repeat the above steps. Bulk import secrets from a file: Create a .yaml file containing the key-value pairs in a flat structure under a "`secrets:` key" per Prepare a File for Bulk Importing Secrets.: For example: `secrets: "MY_API_KEY": "sk-abc123..." "DB_USERNAME": "admin" "DB_PASSWORD": "hunter2" "ENDPOINT_URL": "http://10.0.0.1"` Click Import Yaml File and select the file to upload. To remove all secrets, click Clear All Secrets.
Environment Variables	Environment variables are key–value pairs that you can define for your functions to access at runtime. They are part of the execution environment provided to your functions. For details of how to work with secrets and environment variables, see Working with Secrets and Environment Variables. You will be able to define environment variables per function deployment when you build the pipeline. In this field, define any environment variables that you want to provide to all functions in the pipeline sequence. Function level environment variables overwrite pipeline level ones if the key of a key value pair matches and the value does not. To add environment variables, do one of the following: Enter the variables: Click Add Variable. Enter a key in the Key field and a value in the Value field. If you want to add another variable, click and enter another pair. Repeat until you have added all the variables you want to add. Import environment variables from a file: Prepare the key-value pairs in a .yaml file in a flat structure, per Prepare a File for Bulk Importing Environment Variables. Click Import ConfigMap and select the YAML file to upload.

Click Create Pipeline.
The pipeline resource is created and the Visual Builder opens, enabling you to build the pipeline.

Build the Pipeline

A pipeline can consist of multiple flows (minimally one). Each flow begins with an event trigger. To build the simplest flow, you add a trigger to the pipeline and then connect a function to the trigger. In this case, the pipeline, when deployed, will consume any events that are watched by the chosen trigger's logic, and then invoke the connected function. The function code will usually perform some action on the consumed events.

In your flows, you can create the following connections:

Connect a trigger to a function. In the deployed pipeline, all events consumed by the trigger will invoke the connected function.
Connect a function to another function. In the deployed pipeline, the first function will invoke the second function. You can apply conditions on this connection, so that events are only routed to the connected function if specific conditions apply.

The Visual Builder enables you to add the triggers and functions that you want to build into the pipeline and to connect them to each other. The pipeline can consist of multiple pipeline flows, with triggers invoking functions and functions invoking functions.

Note
You cannot connect a trigger to another trigger. You cannot connect elements in a loop.

During the process, you can click Save Draft to save a draft at any time. When you are done, you can deploy the pipeline.

The following actions are available to help you build the pipeline:

Drag Pipeline Elements into the Builder

Drag and drop triggers and functions into the visual builder to build your pipeline:

To the left of the Visual Builder, select Triggers or Functions to see the library of existing triggers and functions.
To add a trigger or function to the pipeline, drag it into the Visual Builder from the list on the left.

Inspect a Trigger or Function in the Visual Builder

To see the properties of any pipeline element in the visual builder, select the element and click the Inspect button to the right of the builder. The element's properties appear on the right.

For a function, the following information is shown: the container registry where the image is stored, the image source, and the image tag.

For a trigger, the following information is shown: the source view, broker, topic, and the type of trigger.

Search Triggers and Functions

To the left of the Visual Builder, select Triggers or Functions, click the search box and enter a string to search for triggers or functions by name.

Create New Triggers and Functions

To create a new trigger, select the Triggers tab at the left of the Visual Builder and click Create New Trigger.
Follow the procedure described here.
The new trigger is now visible in the library of triggers in the left panel.
To create a new function, select the Functions tab at the left of the Visual Builder and click Create New Function.
Follow the procedure here.

Connect Triggers to Functions and Functions to Functions

To connect a trigger or function to a function, drag the open handle (a) of the first trigger or function to the solid handle (like b) of the function.

A connector appears, showing the direction of invocation (the selected trigger will invoke the selected function):

Apply Event Trigger Labels (Conditional Function Routing)

When you connect one function to another function in a pipeline, you can apply an event trigger label to the connector. This sets conditions for invoking the second function: only events that are marked with the trigger label invoke the connected function.

An event trigger label is a set of one or more key-value pairs. The labels must be applied to events by the invoking function. (See Event Handling for information about authoring your function to set up these labels.)

Connect the invoking function to the function that you want to be invoked conditionally.
The code of the invoking function must be authored to set event trigger labels on events.
Click the route identifier between the functions (the line connecting the two functions).
In the right pane, under Event Trigger Label, enter the label key in the Key field and the label value in the Value field.
A label is added to the connector,
Add more key-value pairs as needed. Events must have all labels that you add in order to invoke the connected function.

Disconnect Elements from Each Other

To disconnect an element from another element, select the arrow and either drag it into the open space or press Backspace or Delete on your keyboard. The link disappears.

Remove Elements from the Builder

To remove a trigger or function from the builder, select the trigger or function and press the BACKSPACE key on your keyboard.

Configure Function Deployment

Each function deployment included in the pipeline has its own configuration that determines the execution environment provided to the function. There is a default configuration for every function deployment. You can edit it for each function deployment as needed.

To edit the configuration a function deployment:

Select (click) the function in the Visual Builder. The function deployment configuration appears on the right.
If you want to change which revision of the function to deploy, select an alternate revision from the Revision Number dropdown.
To edit other parameters, click the Edit button next to the section heading, and edit the values in the fields.
You can configure the following parameters:

Secret Keys

Add secret keys to pass a secret to the specific function deployment. The secret can contain multiple key-value pairs, such as access keys for accessing data and metadata of S3 buckets.

Note
Secret keys that are specified in the pipeline configuration are passed to all function deployments.

You can enable functions to access these secret keys through use of the VAST DataEngine runtime SDK ctx.secrets property in your function code. See The VAST DataEngine Runtime SDK Guide for details.

To create secrets, you can either use the import option and upload a prepared secrets file or enter the secrets.

To enter the secrets:

Select the function in the builder.
Make sure the Settings pane is showing on the right. If not, click Settings.
Scroll down to Secret Keys.
Click create secret.
In the Create Function deployment Secret dialog, do one of the following :
- Click Create Secret and then:
  1. Enter the secret name in the Secret Name field.
  2. In the Key and Value fields provided, enter one of the keys of the secret and a valid value for it.
  3. Click the add button provided to add another key-value pair row as needed until you have added all keys needed for the secret.
  Use the trash can icon to remove a key value pair as you go and the Clear all Secrets button to start over and the Import from Yaml file to import a file for each secret or all secrets.
  To add another secret, click Create Another secret and repeat the previous steps.
- Click Import from YAML to import a prepared secrets file.

Deployment Resources

Most of these resources have minimum and maximum values. Minimum values are guaranteed resource to provide to the function deployment. Maximum values are limits on the maximum resources provided to the function deployment.

Concurrency	The minimum and maximum number of functions to be deployed on the Kubernetes cluster (the number of pods).
CPU	The minimum CPU to guarantee for the function and the maximum CPU to allow to the function.
Memory	The minimum memory to guarantee for the function and the maximum memory to allow to the function.
Autoscaling RPS factor	The rate of requests per second at which autoscaling should begin.
Disk (Ephemeral)	This is not provided by default. If the function needs ephemeral storage, enter an amount of capacity to provide to the function.

Deployment Configuration

Timeout	The timeout for the function.
Retries	The number of retries if the function deployment fails or times out.
Log level	The log level in case of failure. Possible values: NOTSET DEBUG INFO (default) WARNING ERROR CRITICAL
Method of Delivery	Method of processing trigger events. Possible values: ordered. Processes a single event per partition at a time. With this method, the number of concurrent events being processed is limited to the number of topic partitions. This could result in performance limitations. unordered. Events may be processed concurrently.

Batches

These settings configure batch processing for the function. If they are set to default, the function is invoked separately for each event. If you want to accumulate events in batches before invoking the function, set both thresholds. When either threshold is reached, the function will be invoked.

Caution
In order for the batching feature to work, the function code must also be authored to accept and process events in batches. For details of how to write your function code to process events in batches, see Event Handling.
Enabling batches in the DataEngine platform without implementing batch handler code is liable to cause runtime failure.

Important
You must set both thresholds in order to enable batch processing.

Batch Size

A number of events to accumulate before the function is invoked to process those events.

Default: 1. Must be > 0.

Batch Timeout

A period of time to wait until invoking the function. All events accumulated since the last invocation will be passed to the invoked function as a batch.

Enter a number in the field provided. Select the time unit from the dropdown (Secs (seconds) or milliseconds).

Valid range: 0–60000 ms (60 seconds).

Environment Variables

Environment variables are key–value pairs that you can define for your function to access at runtime. They are part of the execution environment provided to your function. Function level environment variables overwrite pipeline level ones if the key of a key value pair matches and the value does not.

For details of how to work with environment variables, see Working with Secrets and Environment Variables.

To add environment variables, do one of the following:

Enter the variables:
1. Click Add Variable.
2. Enter a key in the Key field and a value in the Value field.
3. If you want to add another variable, click and enter another pair.
4. Repeat until you have added all the variables you want to add.
Import environment variables from a file:
1. Prepare the key-value pairs in a .yaml file in a flat structure, per Prepare a File for Bulk Importing Environment Variables.
2. Click Import ConfigMap and select the YAML file to upload.

Deploy the Pipeline

When you are done building a pipeline with the Visual Builder, click Deploy.

To deploy a pipeline that you saved as a draft, find the draft in the Pipelines tab of the Pipelines Management page. The status shows as Draft . Right-click the pipeline and select Deploy.

You can view the status of the pipeline in the Pipelines tab of the Pipelines Management page. Initially, while the pipeline is being deployed, the status is In Progress. When deployed, the status changes to Running.

Documentation Index