Creating a DataEngine Function

Overview

Any function that you want to be able to call and deploy within a DataEngine pipeline must first be packaged as an image and stored on one of the container registries that is connected to the DataEngine instance that you are working with.

VAST provides a capability, through the DataEngine CLI, to package your function code as an image ready to store on a container registry. You then push that image to your container registry. You need to obtain the image tag from the container registry for the specific image.

For each image stored on the container registry, you can create a function resource within DataEngine. A function, in DataEngine terms, is a resource that points to an image of a function stored on a container registry. A function can be built into a DataEngine pipeline and deployed as part of that pipeline. Within a pipeline, a function can be invoked by a trigger or by another function. Each function deployment within a pipeline has a configuration that determines the execution environment provided to the function in that instance.

When you want to update the code for a given function, you can update the code, store the updated image on the container registry and create a new revision of the same function with a different image tag. When you deploy a function in a pipeline, you can choose to deploy a specific revision or the current (latest) revision. When you edit a pipeline, you can switch a function to a different revision.

Workflow

Prepare the Function Image in the Container Registry

Install the VAST DataEngine CLI.

Run the vastde functions init command to create a scaffold for your function.

For example:

 vastde functions init python-pip my-first-function -t ~/functions/

The output is a directory structure needed for implementing your function.

For example:

~/functions/my-first-function/
├── Aptfile          # OS dependencies
├── README.md        # Project documentation
├── customDeps       # custom python libraries
├── main.py          # Handler functions
└── requirements.txt # Python dependencies

0 directories, 5 files

The files in the directory structure are created empty.

Write your code in the files provided to implement the function. See the VAST Cluster 5.5 DataEngine Runtime SDK Developer’s Guide.
The following is sample main.py content:
```
from sharedlib.module import lib

def init(ctx):
    lib.shared_func()

def handler(ctx, event):
    return "Hello World"
```
Run the vastde functions build command to package the function as an image and to tag the image. For example:
```
FUNC_NAME="function"
FUNC_PATH="example/path/to/function"
vastde functions build $FUNC_NAME --target $FUNC_PATH --image-tag my-function
```
Note
This command requires that docker is installed and running locally.
Test the function locally:
1. Run the now built function as a local container: vastde functions localrun.
  Tip
  If your function requires secrets and environment variables, add a file config.yaml to the function scaffold, store the secrets and environment variables in that file for local testing, and run the command with the --config flag.
2. Send a cloud event to the locally running function for testing and invocation: vastde functions invoke.
3. Iterate on code and rebuild as needed.
Tag the local image with a tag of your choice:
```
docker tag ${FUNC_NAME}:TAG CONTAINER_REGISTRY/ARTIFACT_SOURCE:TAG
```
Substitute the following:
- CONTAINER_REGISTRY. The URL of a container registry to which you have privileges to push images. The container registry must also have a connection configured on the tenant for DataEngine. This connection is configured on the VAST Cluster tenant by an administrator.
- ARTIFACT_SOURCE. A path on the container registry where you intend to push the image.
- TAG. The text that you want to tag the image with (such as the name of the function).

Push the image to the container registry:

docker push CONTAINER_REGISTRY/ARTIFACT_SOURCE/${FUNC_NAME}:TAG

Create the Function Resource

From the left navigation menu, select Manage Elements () and then Functions.
Click Create New Function.

Complete the fields:

Function Name	Enter a name for the function.
Description	Enter a description for the function.
Revision Alias	Enter an alias for the initial revision of the function. The alias should be some human-readable, mutable pointer (or label) that targets a specific revision number of your function.
Revision No.	This field is a display field. It displays the revision number, which is initially 1. The number increments on every edit of the function.
Revision Description	Enter a description for the revision.
Container Registry	From the dropdown, select the container registry to which you pushed the image of the function.
Artifact Source	Enter the path to the container image.
Image Tag	Enter the image tag attached to the relevant version of the container image.
Full image path	This is the full path to the image. It is formed automatically from the container registry, artifact source and image tag values that you supply.

Click Create Function.
The function is created and is listed in the Functions page.

Documentation Index

Creating a DataEngine Function

Overview

Workflow

Prepare the Function Image in the Container Registry

Create the Function Resource