Upgrading VAST DataEngine

Caution
Follow these instructions with care to upgrade DataEngine. Following the steps required in the incorrect order for your deployment could result in mismatched component versions and unsuccessful upgrade.

DataEngine Upgrade Stages

Upgrading DataEngine requires three stages that must be performed in the following order to ensure successful upgrade:

VAST Cluster NDU.
If DataEngine is deployed with an external Kubernetes cluster, upgrade the VAST platform and observability services running on the linked external Kubernetes cluster.
Caution
If these services were installed from a VAST Zarf DataEngine package, you must also update them using the same method. This is usually the case if you are upgrading from any release of VAST Cluster 5.4 or if your cluster is deployed in an air-gapped environment.
Upgrade the runtime version of DataEngine function images.

Upgrade the VAST Cluster

Follow Upgrading Your VAST Cluster to upgrade the VAST Cluster to the new version, before you proceed with the next steps.

Upgrade Kubernetes Services

Continue here only after you have upgraded VAST Cluster.

Skip this step if and only if DataEngine is not deployed with an external Kubernetes cluster.

This step upgrades VAST platform and observability services on an external Kubernetes cluster that is linked to DataEngine on any of your cluster's tenants.

Upgrade from Zarf (If VAST Services were Installed from a VAST Zarf DataEngine Package)

This procedure upgrades VAST platform and observability services on the external Kubernetes cluster from a VAST Zarf DataEngine package. This is an offline upgrade method that is suitable for use in air-gapped environments where internet is not accessible.

Do not use any other method to upgrade these services if they were originally installed from a VAST Zarf DataEngine package.

Obtain the VAST DataEngine offline bundle for the installed VAST Cluster version, by contacting VAST Customer Success via a support ticket, Slack, or by sending an email to support@vastdata.com. Be sure to specify your exact VAST Cluster version. For example, "5.5.0-sp2" , since the DataEngine bundle version is tied to the main SW release version. An example file name for this bundle is dataengine_offline-2255166.vast.tar.gz.
Extract the VAST Operator file (for example, zarf-package-dataengine-amd64-1.0.0.tar.zst) and run zarf package deploy with the following options:
```
zarf package deploy --architecture amd64 zarf-package-dataengine-amd64*.tar.zst --confirm --log-level debug
```

Upgrade with VAST DataEngine CLI or Helm Charts (Only if VAST Services were Installed Using Either vastde CLI or Helm Charts)

Caution
Do not use either of these methods to upgrade VAST services on a Kubernetes cluster if the services were originally installed from a VAST Zarf DataEngine package.
Most upgrades from VAST Cluster 5.4 require upgrade from Zarf.
VAST DataEngine CLI supports upgrade of VAST services on a Kubernetes cluster starting from VAST Cluster 5.5.0.

Prerequisites

kubectl access to the external Kubernetes cluster (a working kubeconfig).
Permissions on the target Kubernetes cluster for installing CRDs, RBAC, and a namespace.
Note
Default names for the namespace and the Helm release are vast-dataengine.Override with --namespace and/or --release-name flags if your deployment differs.

Upgrading VAST Services with VAST DataEngine CLI

The vastde upgrade command upgrades VAST services on the Kubernetes cluster using the DataEngine Helm chart, which is embedded in the CLI binary at build time, so the target versioned binary upgrades the CLI locally and enables you to upgrade the services on the Kubernetes cluster.

Download the VAST DataEngine CLI binary for the upgrade target version from the VAST DataEngine CLI Releases page on GitHub.
Grant the binary file execute permissions and move it to your system path.
Confirm that you have installed the desired target vastde CLI version:
```
vastde version
```
Confirm the current release is healthy before you proceed with the upgrade:
```
vastde status
```
Confirm the new binary carries the version you intend to deploy:
```
vastde upgrade --list-versions
```

Run the upgrade:

# Upgrade to the embedded chart version, waiting for readiness
vastde upgrade --wait --timeout 10m

Options:

# Upgrade with additional / changed values
vastde upgrade --set collector.replicas=3

# Reset everything to the new chart's defaults
vastde upgrade --reset-values

# Pin a specific embedded chart version
vastde upgrade --version <chart-version>

# Limit how many release revisions Helm retains (default: 10; 0 = unlimited)
vastde upgrade --history-max 5

Verify:

vastde status
kubectl get pods -n vast-dataengine
kubectl rollout status deployment -n vast-dataengine

Upgrading VAST Services with Helm

To upgrade VAST services from VAST Cluster 5.4.x to 5.5.x:

helm upgrade --install --take-ownership my-de vast/vast-dataengine --reset-then-reuse-values --namespace vast-dataengine --create-namespace

To upgrade VAST services from VAST Cluster 5.5.x to 5.5.x:

helm upgrade my-de vast/vast-dataengine --reset-then-reuse-values

Upgrade the Runtime Version for DataEngine Functions

Continue here only after you have upgraded Kubernetes services.

Run the DataEngine CLI, and make sure it is updated to the latest builder image.

Use vastde builders inspect to check current builder image URL:

user@user-machine:~$ vastde builders inspect
Current Builder Image Configuration

Builder Image URL: https://github.com/vast-data/dataengine-cli/releases/tag/v5.4.1-dev.29d4871e

To change the builder image, use: vastde builders set <image-url>
To view tag details, use: vastde builders inspect <tag>

Run vastde builders list to list available builder images.

Run vastde builders set to update the CLI configuration with the latest builder image URL:

user@user-machine:~$ vastde builders set v5.4.1-dev.c0b8b3d5
Using default repository, full image: vastdataorg/vast-builder:v5.4.1-dev.c0b8b3d5
Builder image URL updated successfully!

Previous: https://github.com/vast-data/dataengine-cli/releases/tag/v5.4.1-dev.c0b8b3d5
Current:  vastdataorg/vast-builder:v5.4.1-dev.29d4871e

This builder image will be used for building DataEngine functions

Rebuild each function and push it to the container registry:
1. Build the function (vastde functions build):
```
vastde functions build my-function --target ./my-function-code --image-tag v2.0
```
2. Tag the local image with a tag of your choice:
```
docker tag my-function:TAG CONTAINER_REGISTRY/ARTIFACT_SOURCE:TAG
```
  Substitute the following:
  - CONTAINER_REGISTRY. The URL of a container registry attached to the DataEngine on the relevant tenant of the VAST Cluster. You need to be logged into the container registry and you need permissions to push images. The container registry must also have a connection configured on the tenant for DataEngine. This connection is configured on the relevant tenant of the VAST Cluster.
  - ARTIFACT_SOURCE. A path on the container registry where you intend to push the image.
  - TAG. The text that you want to tag the image with.
3. Push the image to the container registry:
```
docker push CONTAINER_REGISTRY/ARTIFACT_SOURCE/my-function:TAG
```

Update each function resource to point to its new image. You can do this from the DataEngine GUI:

From the left navigation menu, select Manage Elements () and then Functions.
Right-click the function and select Edit.
The revision number is incremented automatically from the previous revision.
In the Image Tag field, enter the image tag attached to the new version of the container image.

Modify other fields as needed:

Revision Alias	An alias for the revision.
Revision Description	A description specifically for the revision.
Container Registry	The container registry where the image of the function is stored. If different from selected, select the correct container registry from the dropdown.
Artifact Source	The path on the container registry where the image resides.
Full image path	This is the full path to the image. It is formed automatically from the container registry, artifact source and image tag values that you supply.

Click Update Function.
A new revision is created and listed in the Functions page as a revision of the original function.

Redeploy each pipeline with new revisions of its functions:
1. From the left navigation menu, select Manage Pipelines () and then Pipelines.
2. Right-click the pipeline and select Go to Builder.
3. For each function:
  1. Select the function in the builder.
  2. Click Settings to show the function deployment configuration in the Settings pane on the right.
  3. From the Revision Number dropdown, select the new revision of the function.
4. When you are done updating function revisions and making any other changes, click Deploy.
  The pipeline is redeployed.
When some of the pipelines have been redeployed with rebuilt function images, the Dashboard is updated to display new graphs (Active Pipelines, Failing Pipelines and Slowest Pipelines) under the heading System-Wide Overview: Functions And Pipelines Performance And Trends. At this point, the dashboard displays metrics of functions using the newer runtime version. However, metrics from any functions that are still using the old runtime version no longer appear.
When all pipelines are redeployed, metrics from all functions appear. However, the error and the link to this upgrade guide continue to appear while traces from pre-upgrade invocations are still retained.
Once all pipelines are redeployed with rebuilt function images, at least one deployed pipeline has been triggered and the retention period for traces from older functions has transpired, you can navigate to the dashboard (click ) and see the new dashboard fully updated with new data.

Documentation Index