Upgrading Your VAST Cluster

Prev Next

Plan the Upgrade

Prior to upgrade, consider the following:

Upgrade Paths

For an upgrade path to a particular release of  VAST Cluster, see  VAST Cluster release notes for that release.

Note

Before upgrading a VAST on Cloud cluster, you must first upgrade the Multi-Cluster Manager instance that created the VoC cluster. For details of how to upgrade the Multi-Cluster Manager, see VAST on Cloud Overview.VAST on Cloud Overview

Impact and Scheduling

All upgrades are non disruptive (NDU). NDU is done by performing a rolling restart of DNodes and CNodes. On small clusters, this is done one node at a time. On larger clusters, up to 10% of CNodes will be upgraded concurrently, while DNodes will continue to be done one at a time.

The duration of the main upgrade process depends on the cluster's workload and size. Expect a cluster upgrade to take at least one hour for the smallest size cluster (1 DBox and 1CBox) and proportionally longer for larger clusters.

DNode OS upgrade is a lengthy process involving a reboot of each DNode. For non-QLC DBoxes this requires a sequential rebuild of each one of each DNode's NVRAMs. For QLC DBoxes, the duration is much shorter, since NVRAMs are re-mapped from one DNode to another DNode and NVRAMs do not need to be rebuilt.

We recommend not to run any other carrier replacements, server expansions or other such field operations in parallel to the upgrade.

During NDU, clients can continue to write data to the cluster. However, the following operations are blocked:

  • Creating, modifying and deleting virtual IP pools

  • Uploading a support bundle

  • Upgrading, modifying or setting passwords on the cluster

  • Activating or deactivating a CNode or DNode

  • Enabling or disabling an SSD or NVRAM

  • Creating, modifying, removing, refreshing, querying users, generating S3, enabling, disabling and removing S3 access keys for users.

  • Creating, modifying, removing and querying groups

  • Creating S3 replication peers

Check VAST OS Versions (Optional)

Upgrade of VAST OS on the CNodes and DNodes is an optional part of the upgrade procedure. Please confirm with VAST support before enabling this feature.

To check CNode and DNode OS versions in VAST Web UI:

  1. In the VAST Web UI, from the left navigation menu, select Infrastructure and then CNodes or DNodes.

  2. For each host, find the OS listed in the OS column.

To check CNode and DNode OS versions in VAST CLI:

  1. Run cnode list or dnode list.

  2. For each host listed, find the OS listed in the OS-Version column of the command output.

A subset of VAST OS configurations is managed in a new package named vast-utils, which is upgraded during NDUs. The vast-utils package includes cluster networking configurations and host-level services.

The currently installed version of vast-utils is displayed in the VAST-UTILS Version column for each node in the Infrastructure pages in VAST Web UI, as well as in the VAST Utils Version column of cnode list, cnode show, dnode list, dnode show command output in VAST CLI.

Obtain the Target Release Package

Before upgrading, determine how you will obtain the package file for the target release. Target releases are made available several ways:

  • Downloadable from a Microsoft Azure location. These download links are provided in release notes or may be obtained from Support. If you can access Microsoft Azure, you will be able to download target releases from there. You will be able to either:

    • Upload the package from your local machine to VMS. This upload is available through the VAST Web UI as part of the upgrade procedure.

    • Transfer the package to the Management CNode by your own means and then, during the upgrade procedure, specify the path to the package as your upgrade source. This is required if you are upgrading via the VAST CLI and is also supported if you are upgrading via the VAST Web UI.

  • Directly importable from an AWS S3 location. If you can access to AWS, the VAST Web UI upgrade enables you to effect a direct import of a new target release package from an AWS S3 location to the VMS. You may be notified of an AWS S3 link to a new target release within the Upgrade Cluster dialog of the VAST Web UI.

  • If access to these locations is restricted or impractical, contact VAST Data Support for assistance with obtaining target release packages.

Run VAST Support Tool - Upgrade Checks Script

Important

Before upgrading, run the upgrade checks script to verify that the upgrade is expected to proceed without failing.

Beside the mandatory upgrade checks script, there is a VMS feature that enables you to run the same validations that are run at the start of the upgrade process itself, without actually performing the upgrade. This is designed to help you identify an issue that may require fixing so that the upgrade will be successful or to decide to skip the hardware validation stage in the actual upgrade. Should you wish to run this, see Optional: Test Validations.

Optional: Test Validations

The upgrade process performs several validations before the actual upgrade. There is an additional option to run those validations without upgrading before you initiate an upgrade. This is designed to help you identify an issue that may require fixing so that the upgrade will be successful or to decide to skip the hardware validation stage in the actual upgrade.

Test Validations via VAST Web UI

  1. Connect to the VAST Web UI.

  2. From the left navigation menu, select Infrastructure and then Clusters.

  3. Right-click the cluster and select Upgrade.

    The Upgrade Cluster dialog appears.

  4. Click Test Validations.

  5. Review the pre-upgrade validations, select those you want to run and deselect those you want to skip:

    • Hardware Validations.  Validation of hardware component redundancy.

    • Software Validations. Software validations.

    • OS Upgrade Validations.  Validations that are only needed for OS upgrades.

    • Degraded SCM raid.  Validation of the SCM RAID state. If the SCM RAID state is DEGRADED, it means that there is no spare space to rebuild to and that the state will not recover. In an actual upgrade flow, if SCM RAID state is DEGRADED, the upgrade does not proceed unless the force flag is enabled.

  6. Click Test Validations Now.

    The pre-upgrade validations are run. You can monitor the validation tasks from the Activities page. For each validation task, the task is reported done or an error message appears.

Test Validations from the VAST CLI

Run the cluster run-upgrade-validations command. Progress of the tasks and any errors are reported in the command line.

Upgrade via VAST Web UI or VAST CLI

Upgrading to a new build via VAST Web UI

The Upgrade Cluster dialog enables you to do any of the following:

Import a new version without upgrading (you can upgrade later)

  1. Connect to the VAST Web UI.

  2. From the left navigation menu, select Infrastructure and then Clusters.

  3. Right-click the cluster and select Upgrade.

    The Upgrade Cluster wizard appears.

    If no package was previously imported, the wizard opens to the Import Build step. The currently installed build is shown at the top right.

    If a package was imported previously, the wizard opens to the Upgrade step and the details of the package are displayed in the Last Imported Build section of the dialog.

  4. If a package was previously imported but you want to import a new package: either click the Remove build button (import_build_trash_icon.png) to remove the package or click the override button (overwrite_imported_build.png) to import a new package that will overwrite the last imported package.

    The wizard returns to the Import Build step.

  5. From the Import method dropdown, select one of the following to import the target release package:

    • Upload File from your computer. This option lets you upload the target release package from your local machine.

      Browse to the package file that you downloaded, select the file and click Open.

      The file is uploaded. While the file is uploading, the upload progress is shown.

    • Enter URL address. This option lets you enter a path to a target release package from an AWS S3 location.

      Obtain the path from VAST Data Customer Success and paste it into the Cloud URL field in the right pane.

    • Enter VMS Path. If you have transferred the target release package to a location on the CNode that is running VMS, you can use this option to specify the path. The package will be imported from that path.

      Enter the path in the VMS path field in the right pane.

      Note

      If the bundle upload succeeds, the uploaded file will be automatically deleted during post-upgrade cleanup. If the bundle upload fails, no automatic deletion is done. When you no longer need the uploaded file, remove it manually.

  6. Click Import.

    The file is imported.

  7. You can also set configurations for the upgrade without actually upgrading. To do this, review the settings under Review Upgrade Configurations, and adjust as needed:

    • OS+FW. Includes upgrade of VAST OS on the CNodes and firmware on the CNodes and DNodes, as well as the core software build.

      Confirm with VAST Support before enabling this option.

  8. Click Advanced and select any additional options as needed:

    • VMS. Upgrades VMS version only.

    • Update Isolated CPU Config. Resets the configuration of isolated CPUs to an optimal configuration for the cluster according to a formula.

    • BMC Firmware. Includes upgrade of BMC and all relevant components (FPGA, BIOS, UBM, CPLD and so on).

  9. Click Save for Later.

  10. The import begins.

    You can monitor progress in the Activities page.

Import and upgrade now

  1. Connect to the VAST Web UI.

  2. From the left navigation menu, select Infrastructure and then Clusters.

  3. Right-click the cluster and select Upgrade.

    The Upgrade Cluster wizard appears. The currently installed build is shown at the top right.

    If no package has been imported yet, the dialog displays the Import Build step of the wizard.

    If a package has been imported previously, the details of the package are displayed in the Last Imported Build section of the dialog. If the details match the version you wish to upgrade to, click Upgrade Now and skip to step 7.

  4. If a package was previously imported but you want to import a new package: either click the Remove build button (import_build_trash_icon.png) to remove the package or click the override button (overwrite_imported_build.png) to import a new package that will overwrite the last imported package.

    The wizard returns to the Import Build step.

  5. From the Import method dropdown, select one of the following to import the target release package:

    • Upload File from your computer. This option lets you upload the target release package from your local machine.

      Browse to the package file that you downloaded, select the file and click Open.

    • Enter URL address. This option lets you enter a path to a target release package from AWS S3.

      Obtain the path from VAST Data Customer Success and paste it into the Cloud URL field in the right pane.

    • Enter VMS Path. If you have transferred the target release package to a location on the CNode that is running VMS, you can use this option to specify the path. The package will be imported from that path.

      Enter the path in the VMS path field in the right pane.

  6. Click Import and Upgrade Now.

    The file is imported.

  7. Under Review Upgrade Configurations, review and adjust the basic options selection:

    Note

    If no options are selected, only the build is upgraded and the upgrade is non-disruptive.

    • Container Software.

    • OS. Includes upgrade of VAST OS on the CNodes and DNodes as well as the core software build.

  8. Click Advanced and select any additional options as needed:

    • VMS. Upgrades VMS version only.

    • Update Isolated CPU Config. Resets the configuration of isolated CPUs to an optimal configuration for the cluster according to a formula.

    • BMC Firmware. Includes upgrade of BMC and all relevant components (FPGA, BIOS, UBM, CPLD and so on).

      Note

      If the Force flag is set, BMC upgrade is not performed.

  9. Click Upgrade now.

  10. Review the pre-upgrade validations, select those you want to run and deselect those you want to skip:

    • Hardware Validations.  Skips validation of hardware component redundancy.

    • Software Validations. Skips software validations.

    • OS Upgrade Validations.  Skips validations that are only needed for OS upgrades.

    • Degraded SCM raid.  Skips validation of the SCM RAID state. If the SCM RAID state is DEGRADED, it means that there is no spare space to rebuild to and that the state will not recover. In an actual upgrade flow, if SCM RAID state is DEGRADED, the upgrade does not proceed unless the force flag is enabled.

  11. Click Yes, upgrade cluster to confirm the upgrade.

    The release package is now uploaded to the cluster and following upload, the upgrade begins. The Activities icon indicates the upload progress at the top of the page. When the upload is complete, the upgrade begins.

    You can monitor progress in the Activities page.

Upgrade from a pre-imported version

  1. Connect to the VAST Web UI.

  2. From the left navigation menu, select Infrastructure and then Clusters.

  3. Right-click the cluster and select Upgrade.

    The Upgrade Cluster wizard appears. The currently installed build is shown at the top right.

    If a package has been imported previously, the details of the package are displayed in the Last Imported Build section of the dialog.

  4. Click Upgrade Now.

  5. Under Review Upgrade Configurations, review and adjust the basic options selection:

    Note

    If no options are selected, only the build is upgraded and the upgrade is non-disruptive.

    • Container Software.

    • OS. Includes upgrade of VAST OS on the CNodes and DNodes as well as the core software build.

  6. Click Advanced and select any additional options as needed:

    • VMS. Upgrades VMS version only.

    • Update Isolated CPU Config. Resets the configuration of isolated CPUs to an optimal configuration for the cluster according to a formula.

    • BMC Firmware. Includes upgrade of BMC and all relevant components (FPGA, BIOS, UBM, CPLD and so on).

  7. Click Yes, upgrade cluster to confirm the upgrade.

    The upgrade begins. The Activities icon indicates the upload progress at the top of the page. When the upload is complete, the upgrade begins.

    You can monitor progress in the Activities page.

Upgrading via VAST CLI

  1. Connect to the management CNode using the local IP (not the VMS virtual IP).

  2. Transfer the package file to the management CNode, placing it under /vast/bundles. For example, if you are provided with a link to download the package from Azure:

    wget link -O /vast/bundles/filename

    (where link refers to the full download link, and filename refers to the name of the release package file.)

    Note

    It is important that the file is specifically copied to /vast/bundles and not to any other location.

  3. Run the following command to start VAST CLI:

    vcli -u <username> -p <password>
    
  4. From the VAST CLI run the cluster upgrade command.

    For example, for NDU of the software build alone:

    cluster upgrade --build-package /vast/bundles/release-<release number>-<pipeline number>.vast.tar.gz

    Alternatively, with VAST OS upgrade on the CNodes and DNodes:

    cluster upgrade --build-package /vast/bundles/release-<release number>-<pipeline number>.vast.tar.gz --os-upgrade
  5. Wait patiently for upgrade to complete.

    Initially, the build file is extracted and imported, which takes some time. Once that stage is complete, the upgrade begins and the progress is displayed in the CLI.