Plan the Upgrade
Prior to upgrade, consider the following:
Upgrade Paths
For an upgrade path to a particular release of VAST Cluster, see VAST Cluster release notes for that release.
Note
Before upgrading a VAST on Cloud cluster, you must first upgrade the Multi-Cluster Manager instance that created the VoC cluster. For details of how to upgrade the Multi-Cluster Manager, see VAST on Cloud Overview.
Impact and Scheduling
All upgrades are non disruptive (NDU). NDU is done by performing a rolling restart of DNodes and CNodes. On small clusters, this is done one node at a time. On larger clusters, up to 10% of CNodes will be upgraded concurrently, while DNodes will continue to be done one at a time.
The duration of the main upgrade process depends on the cluster's workload and size. Expect a cluster upgrade to take at least one hour for the smallest size cluster (1 DBox and 1CBox) and proportionally longer for larger clusters.
DNode OS upgrade is a lengthy process involving a reboot of each DNode. For non-QLC DBoxes this requires a sequential rebuild of each one of each DNode's NVRAMs. For QLC DBoxes, the duration is much shorter, since NVRAMs are re-mapped from one DNode to another DNode and NVRAMs do not need to be rebuilt.
We recommend not to run any other carrier replacements, server expansions or other such field operations in parallel to the upgrade.
During NDU, clients can continue to write data to the cluster. However, the following operations are blocked:
Creating, modifying and deleting virtual IP pools
Uploading a support bundle
Upgrading, modifying or setting passwords on the cluster
Activating or deactivating a CNode or DNode
Enabling or disabling an SSD or NVRAM
Creating, modifying, removing, refreshing, querying users, generating S3, enabling, disabling and removing S3 access keys for users.
Creating, modifying, removing and querying groups
Creating S3 replication peers
Check VAST OS Versions (Optional)
Upgrade of VAST OS on the CNodes and DNodes is an optional part of the upgrade procedure. Please confirm with VAST Customer Success before enabling this feature.
In the VAST Web UI, from the left navigation menu, select Infrastructure and then CNodes or DNodes.
For each host, find the OS listed in the OS column.
Run
cnode listordnode list.For each host listed, find the OS listed in the
OS-Versioncolumn of the command output.
Starting with VAST Cluster 5.3.2, a subset of VAST OS configurations is managed in a new package named vast-utils, which is upgraded during NDUs. The vast-utils package includes cluster networking configurations and host-level services.
The currently installed version of vast-utils is displayed in the VAST-UTILS Version column for each node in the Infrastructure pages in VAST Web UI, as well as in the VAST Utils Version column of cnode list, cnode show, dnode list, dnode show command output in VAST CLI.
Obtain the Target Release Package
Before upgrading, determine how you will obtain the package file for the target release. Target releases are made available several ways:
Downloadable from a Microsoft Azure location. These download links are provided in release notes or may be obtained from Support. If you can access Microsoft Azure, you will be able to download target releases from there. You will be able to either:
Upload the package from your local machine to VMS. This upload is available through the VAST Web UI as part of the upgrade procedure.
Transfer the package to the Management CNode by your own means and then, during the upgrade procedure, specify the path to the package as your upgrade source. This is required if you are upgrading via the VAST CLI and is also supported if you are upgrading via the VAST Web UI.
Directly importable from an AWS S3 location. If you can access to AWS, the VAST Web UI upgrade enables you to effect a direct import of a new target release package from an AWS S3 location to the VMS. You may be notified of an AWS S3 link to a new target release within the Upgrade Cluster dialog of the VAST Web UI.
If access to these locations is restricted or impractical, contact VAST Data Support for assistance with obtaining target release packages.
Run Pre-Upgrade Validation (Optional)
The upgrade process performs several validations before the actual upgrade. There is an additional option to run validations without upgrading before you initiate an upgrade. This can help you identify an issue that may require fixing so that the upgrade will be successful or to decide to skip the hardware validation stage in the actual upgrade.
Running Pre-Upgrade Validation via VAST Web UI
Connect to the VAST Web UI.
From the left navigation menu, select Infrastructure and then Clusters.
Right-click the cluster and select Upgrade.
The Upgrade Cluster dialog appears.
Click Test Validations.
Review the pre-upgrade validations, select those you want to run and deselect those you want to skip:
Hardware Validations. Validation of hardware component redundancy.
Software Validations. Software validations.
OS Upgrade Validations. Validations that are only needed for OS upgrades.
Degraded SCM raid. Validation of the SCM RAID state. If the SCM RAID state is DEGRADED, it means that there is no spare space to rebuild to and that the state will not recover. In an actual upgrade flow, if SCM RAID state is DEGRADED, the upgrade does not proceed unless the force flag is enabled.
Click Test Validations Now.
The pre-upgrade validations are run. You can monitor the validation tasks from the Activities page. For each validation task, the task is reported done or an error message appears.
Running Pre-Upgrade Validation from the VAST CLI
Run the cluster run-upgrade-validations command. Progress of the tasks and any errors are reported in the command line.
Upgrade via VAST Web UI or VAST CLI
Upgrading via VAST CLI (requires manually transferring the package file to the management CNode.)
Upgrading to a new build via VAST Web UI
The Upgrade Cluster dialog enables you to do any of the following:
Import a new version without upgrading (you can upgrade later)
Connect to the VAST Web UI.
From the left navigation menu, select Infrastructure and then Clusters.
Right-click the cluster and select Upgrade.
The Upgrade Cluster wizard appears.
If no package was previously imported, the wizard opens to the Import Build step. The currently installed build is shown at the top right.
If a package was imported previously, the wizard opens to the Upgrade step and the details of the package are displayed in the Last Imported Build section of the dialog.
If a package was previously imported but you want to import a new package: either click the Remove build button (
) to remove the package or click the override button (
) to import a new package that will overwrite the last imported package. The wizard returns to the Import Build step.
Notice
The override button is introduced in VAST Cluster 5.3.1.
From the Import method dropdown, select one of the following to import the target release package:
Upload File from your computer. This option lets you upload the target release package from your local machine.
Browse to the package file that you downloaded, select the file and click Open.
The file is uploaded. While the file is uploading, the upload progress is shown.
Enter URL address. This option lets you enter a path to a target release package from an AWS S3 location.
Obtain the path from VAST Data Customer Success and paste it into the Cloud URL field in the right pane.
Enter VMS Path. If you have transferred the target release package to a location on the CNode that is running VMS, you can use this option to specify the path. The package will be imported from that path.
Enter the path in the VMS path field in the right pane.
Note
If the bundle upload succeeds, the uploaded file will be automatically deleted during post-upgrade cleanup. If the bundle upload fails, no automatic deletion is done. When you no longer need the uploaded file, remove it manually.
Click Import.
The file is imported.
You can also set configurations for the upgrade without actually upgrading. To do this, review the settings under Review Upgrade Configurations, and adjust as needed:
OS+FW. Includes upgrade of VAST OS on the CNodes and firmware on the CNodes and DNodes, as well as the core software build.
Confirm with VAST Support before enabling this option.
Click Advanced and select any additional options as needed:
VMS. Upgrades VMS version only.
Update Isolated CPU Config. Resets the configuration of isolated CPUs to an optimal configuration for the cluster according to a formula.
BMC Firmware. Includes upgrade of BMC and all relevant components (FPGA, BIOS, UBM, CPLD and so on).
Click Save for Later.
The import begins.
You can monitor progress in the Activities page.
Import and upgrade now
Connect to the VAST Web UI.
From the left navigation menu, select Infrastructure and then Clusters.
Right-click the cluster and select Upgrade.
The Upgrade Cluster wizard appears. The currently installed build is shown at the top right.
If no package has been imported yet, the dialog displays the Import Build step of the wizard.
If a package has been imported previously, the details of the package are displayed in the Last Imported Build section of the dialog. If the details match the version you wish to upgrade to, click Upgrade Now and skip to step 7.
If a package was previously imported but you want to import a new package: either click the Remove build button (
) to remove the package or click the override button (
) to import a new package that will overwrite the last imported package. The wizard returns to the Import Build step.
Notice
The override button is introduced in VAST Cluster 5.3.1.
From the Import method dropdown, select one of the following to import the target release package:
Upload File from your computer. This option lets you upload the target release package from your local machine.
Browse to the package file that you downloaded, select the file and click Open.
Enter URL address. This option lets you enter a path to a target release package from AWS S3.
Obtain the path from VAST Data Customer Success and paste it into the Cloud URL field in the right pane.
Enter VMS Path. If you have transferred the target release package to a location on the CNode that is running VMS, you can use this option to specify the path. The package will be imported from that path.
Enter the path in the VMS path field in the right pane.
Click Import and Upgrade Now.
The file is imported.
Under Review Upgrade Configurations, review and adjust the basic options selection:
Note
If no options are selected, only the build is upgraded and the upgrade is non-disruptive.
Container Software.
OS. Includes upgrade of VAST OS on the CNodes and DNodes as well as the core software build.
Click Advanced and select any additional options as needed:
VMS. Upgrades VMS version only.
Update Isolated CPU Config. Resets the configuration of isolated CPUs to an optimal configuration for the cluster according to a formula.
BMC Firmware. Includes upgrade of BMC and all relevant components (FPGA, BIOS, UBM, CPLD and so on).
Click Upgrade now.
Review the pre-upgrade validations, select those you want to run and deselect those you want to skip:
Hardware Validations. Skips validation of hardware component redundancy.
Software Validations. Skips software validations.
OS Upgrade Validations. Skips validations that are only needed for OS upgrades.
Degraded SCM raid. Skips validation of the SCM RAID state. If the SCM RAID state is DEGRADED, it means that there is no spare space to rebuild to and that the state will not recover. In an actual upgrade flow, if SCM RAID state is DEGRADED, the upgrade does not proceed unless the force flag is enabled.
Click Yes, upgrade cluster to confirm the upgrade.
The release package is now uploaded to the cluster and following upload, the upgrade begins. The Activities icon indicates the upload progress at the top of the page. When the upload is complete, the upgrade begins.
You can monitor progress in the Activities page.
Upgrade from a pre-imported version
Connect to the VAST Web UI.
From the left navigation menu, select Infrastructure and then Clusters.
Right-click the cluster and select Upgrade.
The Upgrade Cluster wizard appears. The currently installed build is shown at the top right.
If a package has been imported previously, the details of the package are displayed in the Last Imported Build section of the dialog.
Click Upgrade Now.
Under Review Upgrade Configurations, review and adjust the basic options selection:
Note
If no options are selected, only the build is upgraded and the upgrade is non-disruptive.
Container Software.
OS. Includes upgrade of VAST OS on the CNodes and DNodes as well as the core software build.
Click Advanced and select any additional options as needed:
VMS. Upgrades VMS version only.
Update Isolated CPU Config. Resets the configuration of isolated CPUs to an optimal configuration for the cluster according to a formula.
BMC Firmware. Includes upgrade of BMC and all relevant components (FPGA, BIOS, UBM, CPLD and so on).
Click Yes, upgrade cluster to confirm the upgrade.
The upgrade begins. The Activities icon indicates the upload progress at the top of the page. When the upload is complete, the upgrade begins.
You can monitor progress in the Activities page.
Upgrading via VAST CLI
Connect to the management CNode using the local IP (not the VMS virtual IP).
Transfer the package file to the management CNode, placing it under
/vast/bundles. For example, if you are provided with a link to download the package from Azure:wget link -O /vast/bundles/filename
(where link refers to the full download link, and filename refers to the name of the release package file.)
Note
It is important that the file is specifically copied to
/vast/bundlesand not to any other location.Run the following command to start VAST CLI:
vcli -u <username> -p <password>
From the VAST CLI run the cluster upgrade command.
For example, for NDU of the software build alone:
cluster upgrade --build-package /vast/bundles/release-<release number>-<pipeline number>.vast.tar.gz
Alternatively, with VAST OS upgrade on the CNodes and DNodes:
cluster upgrade --build-package /vast/bundles/release-<release number>-<pipeline number>.vast.tar.gz --os-upgrade
Wait patiently for upgrade to complete.
Initially, the build file is extracted and imported, which takes some time. Once that stage is complete, the upgrade begins and the progress is displayed in the CLI.