Enabling DataEngine on a VAST Cluster Tenant

Establish prerequisite services.

Configure Prerequisites on the VAST Cluster. These are configurations that you must be established on the VAST Cluster tenant before you enable DataEngine.

Enable DataEngine. In this step, you will be prompted to connect the prerequisite external services to the tenant.

The first time you use the Kubernetes cluster to deploy DataEngine on any tenant on the VAST cluster, you need to install VAST DataEngine platform and observability services on the Kubernetes cluster, to make sure that DataEngine is provisioned correctly on the cluster. Install the VAST Zarf DataEngine package on each Kubernetes cluster in your environment that you intend to use to deploy VAST DataEngine functions and pipelines.

There are three ways of installing these services:

For any DataEngine deployment on any given tenant, create a unique namespace on the Kubernetes cluster. It is good practice to use a consistent naming convention that includes the name of the tenant in the name of the namespace. This will help a cluster admin or tenant admin user to identify the dedicated tenant specific namespace when linking the Kubernetes cluster and selecting a namespace.

Note

Sharing namespaces across tenants is not supported.

Download the VAST DataEngine CLI binary for the upgrade target version from the VAST DataEngine CLI Releases page on GitHub).

Grant the binary file execute permissions and move it to your system path.

Confirm that you have installed the desired target vastde CLI version:

vastde version

Confirm the current release is healthy before you proceed with the upgrade:

vastde status

Preview the install - render the manifests without applying anything:

vastde install --dry-run

To see only the non-default Helm values that would be applied:

vastde install --dry-run-helm-values

Confirm the new binary carries the version you intend to deploy:

vastde upgrade --list-versions

Install:

# Install with default settings into the vast-dataengine namespace
vastde install

# Install and block until all resources are Ready (recommended)
vastde install --wait --timeout 10m

Options:

# Custom values file
vastde install -f my-values.yaml

# Inline value overrides
vastde install --set collector.replicas=3

# Custom namespace (created automatically unless --create-namespace=false)
vastde install --namespace my-de-namespace

# Pin a specific embedded chart version (default: latest embedded)
vastde install --version <chart-version>

Verify:

vastde status
kubectl get pods -n vast-dataengine
kubectl rollout status deployment -n vast-dataengine

Run the following to add the repository (where GH_USER is a GitHub user with access to the vast-data org):

GH_USER=<GH_USER>
TOKEN=$(gh auth token --user $GH_USER)
helm repo add vast https://raw.githubusercontent.com/vast-data/charts/main \
  --username $GH_USER --password "$TOKEN"
helm repo update vast

Run the following to install the charts:

helm search repo vast/vast-dataengine
helm pull vast/vast-dataengine --devel --untar       # for inspection
helm template my-de vast/vast-dataengine -f values.yaml
helm install  my-de vast/vast-dataengine -f values.yaml \
  --namespace vast-dataengine --create-namespace

Ensure you have a valid Kubeconfig file for connecting to the Kubernetes cluster.

Verify that the inotify limits are large enough to allow for VAST Zarf DataEngine package installation:

Obtain the VAST DataEngine offline bundle for the installed VAST Cluster version, by contacting VAST Customer Success via a support ticket, Slack, or by sending an email to support@vastdata.com. Be sure to specify your exact VAST Cluster version. For example, "5.4.3-sp2" , since the DataEngine bundle version is tied to the main SW release version. An example file name for this bundle is dataengine_offline-2255166.vast.tar.gz.

Extract the Zarf agent init package from the bundle (for example zarf-init-amd64-v0.60.0.tar.zst) and run zarf init with the following options :

zarf init --architecture amd64 zarf-init-amd64-v0.60.0.tar.zst --set REGISTRY_HPA_AUTO_SIZE=true --confirm --log-level debug

Note

The above call assumes that a default storage class exists. Otherwise, add the --storage-class option to the call. For example --storage-class=local-path.

Extract the VAST Operator file (for example, zarf-package-dataengine-amd64-1.0.0.tar.zst) and run zarf package deploy with the following options:

zarf package deploy --architecture amd64 zarf-package-dataengine*.tar.zst --confirm --log-level debug

Ensure you have a valid Kubeconfig file for connecting to the Kubernetes cluster.

Verify that the inotify limits are large enough to allow for VAST Zarf DataEngine package installation:

Obtain the VAST DataEngine offline bundle for the installed VAST Cluster version, by contacting VAST Customer Success via a support ticket, Slack, or by sending an email to support@vastdata.com. Be sure to specify your exact VAST Cluster version. For example, "5.4.3-sp2" , since the DataEngine bundle version is tied to the main SW release version. An example file name for this bundle is dataengine_offline-2255166.vast.tar.gz.

Extract the VAST Operator file (for example, zarf-package-dataengine-amd64-1.0.0.tar.zst) and run zarf package deploy with the following options:

zarf package deploy --architecture amd64 zarf-package-dataengine-amd64*.tar.zst --confirm --log-level debug

From the left navigation menu, select Element Store and then Tenants.

Click Create Tenant to open the Create New Tenant dialog.

In the General tab, complete the fields:

Click Create. The tenant is created and appears in the list of tenants in the Tenants page.

Make sure the compute cluster exists.

If you need to create one, see Creating a VAST Compute Cluster.

Do one of the following to attach the compute cluster to the tenant:

Go to Settings -> Notifications -> Notification Kafka Brokers.

Click Add a New Kafka Broker and complete these fields:

Tip

Ensure that the hosts are accessible from the VAST cluster's management interface at the specified ports.

Click Add Kafka Broker. 

The newly created event broker configuration is added to the list of event brokers.

Make sure the provider you want to use is attached to the tenant. The provider can be a VAST provider, LDAP, or Active Directory. The provider should be configured on the cluster. For general information about managing providers, see Providers.

Note

If you use a VAST provider, enable the provider for cluster admin users and tenant admin users to manage and view, as needed. The default VAST provider is initially enabled for management by cluster admin users only.

To attach a provider to a tenant:

Create a user on the provider. Grant the user bucket create and bucket delete permissions.

Tip

It is usually best practice that the user belongs to a group designated for granting DataEngine access. You will be able to grant the user's group bucket listing permission for a VAST Event Broker and you will be able to grant the same group access to DataEngine.

Tip

To create a user as part of a group on a VAST provider:

1. From the left navigation menu, select User Management and then, under Users and Groups, Local Groups.

2. On the Groups page that opens, click + Create Group.

3. In the Add Group dialog, complete the fields:

   Field	Description
   Name (required)	Enter a name for the group.
   GID (required)	Enter a POSIX group ID (GID) for the group.
   VAST Provider (required)	Select the VAST provider with which the group will be associated.

4. Click Create. The group is created.

5. Switch to the Local Users tab.

6. Click Create User and complete the following fields:

   Field	Description
   Name (required)	The user name.
   UID	The user's POSIX UID.
   VAST Provider (required)	Select the VAST provider with which the user will be associated.
   Leading group	The name of the user's leading group. This is the group assigned by default as the owning group of any files created by the user. Select the group you created in the previous step from the dropdown.
   Groups	Names of other groups that the user belongs to beside the leading group. Also known as auxiliary groups. Select groups from the dropdown. If a group has not been added to the VAST provider, add the group first.

Under Permissions Access, select the tenant in the Select tenant to see user details list. Tenants associated with the selected VAST provider (if any) are shown, as well as the default tenant.

Enable bucket create and delete permissions:

Tip

Alternatively, you can grant these permissions through an identity policy attached to either the group or the user.

Do one of the following, as appropriate, to grant the user an S3 access key pair:

In the VAST Web UI, select Element Store from the left navigation menu and then select Views.

Click Create View to add a new view.

The Add View dialog appears.

Complete the following fields:

Create or select a suitable view policy. The view policy needs to have S3 Native security flavor enabled. The policy should also grant S3 bucket listing permission to the user group that you are using to provision application users, although you could add the bucket listing permission at a later stage. Bucket listing permission will enable all group members to create triggers using the event broker you are creating.

Create or select a virtual IP pool for the event broker. The virtual IP pool needs to include at least as many virtual IP addresses as there are CNodes on the cluster. It also needs to be dedicated exclusively to the VAST Event Broker.

Click Create.

From the left navigation menu, select Element Store and then Tenants.

Right-click the tenant and select Enable DataEngine.

The Enable DataEngine wizard opens.

Note

If the wizard closes at any stage for any reason, you can resume the wizard.

In the Assign Kafka step screen, from the Default Broker dropdown, do one of the following:

From the Default Topic dropdown, select Add new topic to create the default topic on the event broker. Triggers will be streamed to this topic by default and it will be available immediately for hosting triggers when DataEngine is enabled. In most cases, there is no need to create additional topics for hosting triggers. Assuming the default topic will be used for all or most triggers, it's advisable to set wide partitioning, probably in the region of 50-100 partitions.

In the Create Topic dialog, complete the following required fields:

Click Create.

In the Default Deadletter Topic dropdown, select Add new topic to create the deadletter topic, which is the topic to which events are routed in case of function failures, routing failures and dispatching failures. The deadletter topic will not be available to configure as the topic for any trigger event.

Click Add Certificate to upload a TLS certificate.

Paste a CA certificate into the box provided and click Upload Certificate.

Click Next.

The Assign Compute Cluster stage starts.

Click Click Here ("to assign a tenant to an existing cluster").

The wizard closes and the Compute Clusters tab of the Data Engine page appears.

From the Tenants dropdown, select All VAST Compute Clusters.

All VAST compute clusters on the VAST cluster are listed.

Identify the VAST compute cluster that you want to link to DataEngine.

If needed, create a new one.

Right-click the compute cluster, select Attach Tenant.

The Attach Tenants To Compute Cluster dialog appears.

From the Tenants dropdown, select each tenant you want to add and click Add Tenants.

Click Save.

The VAST compute cluster is now attached to the tenant and linked to DataEngine on the tenant.

Resume the Enable DataEngine wizard:

If you want to link an external compute cluster as well as the VAST compute cluster, continue with Link an External Compute Cluster. Otherwise, skip to Link a Container Registry (All Deployments).

In the Assign Compute Cluster step, if the Assign external compute cluster checkbox appears, enable the checkbox and click Next.

In the Assign External Compute Cluster screen, complete these fields:

In the Certificates area, use the fields provided to upload the CA certificate, client certificate and private key used for authenticating over mTLS to the compute cluster. For each certificate, click +Add new , paste the certificate content, base64-encoded, and then click Save Client certificate /Save CA certificate/Save Private Key.

In the Tags area, enter any metadata tag that you want to tag the compute cluster resource with, and select Add Tag to add the tag. Repeat for additional tags. This is optional.

Click Next.

In the Select Namespaces step, the namespaces on the compute cluster are listed on the left under All possible Properties. Use the selection boxes and arrow buttons to choose which of the namespaces to allow the tenant's DataEngine to access and list them on the right under Selected Properties. Pipelines will be deployed in the selected namespace(s). Choose a unique namespace for the tenant. Namespaces cannot be shared by multiple tenants.

Caution

Do not select Zarf-installed workspaces such as vast-dataengine, knative-eventing, and knative-servering. Using Zarf-owned namespaces workspaces for pipeline deployments will fail since pods will fail to pull function images from the correct location.

Click Next.

In the Link Container Registry step, complete the fields to connect the container registry:

Click Finish.

DataEngine is now enabled on the tenant with a compute cluster and a container registry. Proceed to provisioning users.