Upgrading to VAST Cluster 5.2

Prev Next

Plan the Upgrade

Prior to upgrade, consider the following:

Upgrade Paths

For an upgrade path to a particular release of  VAST Cluster, see  VAST Cluster release notes for that release.

Note

Before upgrading a VAST on Cloud cluster, you must first upgrade the Multi-Cluster Manager instance that created the VoC cluster. For details of how to upgrade the Multi-Cluster Manager, see Upgrading VoC Clusters.

Impact and Scheduling

All upgrades are non disruptive (NDU). NDU is done by performing a rolling restart of DNodes and CNodes. On small clusters, this is done one node at a time. On larger clusters, up to 10% of CNodes will be upgraded concurrently, while DNodes will continue to be done one at a time.

The duration of the main upgrade process depends on the cluster's workload and size. Expect a cluster upgrade to take at least one hour for the smallest size cluster (1 DBox and 1CBox) and proportionally longer for larger clusters.

DNode OS upgrade is a lengthy process involving a reboot of each DNode. For non-QLC DBoxes this requires a sequential rebuild of each one of each DNode's NVRAMs. For QLC DBoxes, the duration is much shorter, since NVRAMs are re-mapped from one DNode to another DNode and NVRAMs do not need to be rebuilt.

We recommend not to run any other carrier replacements, server expansions or other such field operations in parallel to the upgrade.

During NDU, clients can continue to write data to the cluster. However, the following operations are blocked:

  • Creating, modifying and deleting virtual IP pools

  • Uploading a support bundle

  • Upgrading, modifying or setting passwords on the cluster

  • Activating or deactivating a CNode or DNode

  • Enabling or disabling an SSD or NVRAM

  • Creating, modifying, removing, refreshing, querying users, generating S3, enabling, disabling and removing S3 access keys for users.

  • Creating, modifying, removing and querying groups

  • Creating S3 replication peers

Check VAST OS Versions (Optional)

Upgrade of VAST OS on the CNodes and DNodes is an optional part of the upgrade procedure. Confirm with VAST Support before enabling this feature.

To check CNode and DNode OS versions via VAST Web UI:

  1. In the VAST Web UI, from the left navigation menu, select Infrastructure and then CNodes or DNodes.

  2. For each host, find the OS listed in the OS column.

To check CNode and DNode OS versions via VAST CLI:

  1. Run cnode list or dnode list.

  2. For each host listed, find the OS listed in the OS-Version column of the command output.

Starting with VAST Cluster 5.2.2, a subset of VAST OS configurations is managed in a new package named vast-utils, which is upgraded during NDUs. The vast-utils package includes cluster networking configurations and host-level services.

The currently installed version of vast-utils is displayed in the VAST-UTILS Version column for each node in the Infrastructure pages in VAST Web UI, as well as in the VAST Utils Version column of cnode list, cnode show, dnode list, dnode show command output in VAST CLI.

Obtain the Target Release Package

Before upgrading, determine how you will obtain the package file for the target release. Target releases are made available several ways:

  • Downloadable from a Microsoft Azure location. These download links are provided in release notes or may be obtained from Support. If you can access Microsoft Azure, you will be able to download target releases from there. You will be able to either:

    • Upload the package from your local machine to VMS. This upload is available through the VAST Web UI as part of the upgrade procedure.

    • Transfer the package to the Management CNode by your own means and then, during the upgrade procedure, specify the path to the package as your upgrade source. This is required if you are upgrading via the VAST CLI and is also supported if you are upgrading via the VAST Web UI.

  • Directly importable from an AWS S3 location. If you can access to AWS, the VAST Web UI upgrade enables you to effect a direct import of a new target release package from an AWS S3 location to the VMS. You may be notified of an AWS S3 link to a new target release within the Upgrade Cluster dialog of the VAST Web UI.

  • If access to these locations is restricted or impractical, contact VAST Data Support for assistance with obtaining target release packages.

Run Pre-Upgrade Validation (Optional)

The upgrade process performs several validations before the actual upgrade. There is an additional option to run validations without upgrading before you initiate an upgrade. This can help you identify an issue that may require fixing so that the upgrade will be successful or to decide to skip the hardware validation stage in the actual upgrade.

Running Pre-Upgrade Validation via VAST Web UI

  1. Connect to the VAST Web UI.

  2. From the left navigation menu, select Infrastructure and then Clusters.

  3. Open the Actions menu for the cluster and select Upgrade.

    The Upgrade Cluster dialog appears.

  4. Go to the Upgrade Validations tab and set options to run or skip pre-upgrade validations:

    • Skip Hardware Validations.  Skips validation of hardware component redundancy.

    • Skip Software Validations. Skips software validations.

    • Skip OS Upgrade Validations.  Skips validations that are only needed for OS upgrades.

    • Skip Degraded SCM raid.  Skips validation of the SCM RAID state. If the SCM RAID state is DEGRADED, it means that there is no spare space to rebuild to and that the state will not recover. In an actual upgrade flow, if SCM RAID state is DEGRADED, the upgrade does not proceed unless the force flag is enabled.

  5. Click Run Validation.

    The pre-upgrade validations are run. You can monitor the validation tasks from the Activities page. For each validation task, the task is reported done or an error message appears.

Running Pre-Upgrade Validation from the VAST CLI

Run the cluster run-upgrade-validations command. Progress of the tasks and any errors are reported in the command line.

Upgrade via VAST Web UI or VAST CLI

Upgrading to a new build via VAST Web UI

The Upgrade Cluster dialog enables you to do any of the following:

Import a new version without upgrading (you can upgrade later)

  1. Connect to the VAST Web UI.

  2. From the left navigation menu, select Infrastructure and then Clusters.

  3. Open the Actions menu for the cluster and select Upgrade.

    The Upgrade Cluster dialog appears with the Import New Build tab open. The currently installed build is shown at the bottom left.

  4. In the Import New Build tab, select one of the following to import the target release package:

    • Via Notification (default). A notification of a new available version may appear on this tab. You can use this as an import source if you have AWS access. When this source is selected, you do not need to enter any values in the right pane.

    • Local File. This option lets you upload the target release package from your local machine.

      In the right pane, click Upload a file and browse to the package file that you downloaded.

    • Cloud. This option lets you enter a path to a target release package that is available on AWS S3.

      Obtain the path from VAST Data Support and paste it into the Cloud URL field in the right pane.

    • VMS Path. If you have transferred the target release package to a location on the CNode that is running VMS, you can use this option to specify the path. The package will be imported from that path.

      Enter the path in the VMS path field in the right pane.

  5. Click Import.

  6. In Import a new version?, optionally enable Skip prepare, which skips the following:

    • Pre-upgrade validations

    • Copying the package to other hosts

    • Pulling the image in all CNodes.

  7. Click Yes, import a new version.

    The import begins.

    You can monitor progress in the Activities page.

Import and upgrade now

  1. Connect to the VAST Web UI.

  2. From the left navigation menu, select Infrastructure and then Clusters.

  3. Open the Actions menu for the cluster and select Upgrade.

    The Upgrade Cluster dialog appears, with the currently installed build shown at the bottom left.

  4. If you are not sure if the target release package was already imported, go to the Upgrade to Last Imported Build tab. If a package has been imported previously, the details of the package are displayed there. If the details match the version you wish to upgrade to, click Upgrade and skip to step 7.

  5. If the target release package has not been imported yet, in the Import New Build tab, select one of the following to import the target release package:

    • Via Notification (default). A notification of a new available version may appear on this tab. You can use this as an import source if you have AWS access. When this source is selected, you do not need to enter any values in the right pane.

    • Local File. This option lets you upload the target release package from your local machine.

      In the right pane, click Upload a file and browse to the package file that you downloaded.

    • Cloud. This option lets you enter a path to a target release package that is available on AWS S3.

      Obtain the path from VAST Data Support and paste it into the Cloud URL field in the right pane.

    • VMS Path. If you have transferred the target release package to a location on the CNode that is running VMS, you can use this option to specify the path. The package will be imported from that path.

      Enter the path in the VMS path field in the right pane.

  6. Click Import and Upgrade Now.

  7. In Upgrade cluster to new version?, carefully set the options you need.

    Note

    If no options are selected, only the build is upgraded and the upgrade is non-disruptive.

    • OS Upgrade. Includes upgrade of VAST OS on the CNodes and DNodes as well as the core software build.

    • BMC Upgrade. Includes upgrade of BMC and all relevant components (FPGA, BIOS, UBM, CPLD and so on).

    • Isolate CPUs. Resets the configuration of isolated CPUs to an optimal configuration for the cluster according to a formula.

    • Force. Forces upgrade regardless of version or upgrade state. Use this option if you need to upgrade to the same version as already installed or if a previous upgrade failed without the upgrade state clearing.

      Caution

      Select this option only if directed explicitly by VAST Data Support.

    • FW Upgrade. For CERES, upgrades NIC firmware for BMC, MCU, PCI and Bluefield.

    • DR. Enables data reduction on a cluster that does not have data reduction enabled before upgrade.

    • Vms-only upgrade. Upgrades VMS version only.

  8. Click Yes, upgrade cluster.

  9. Review the upgrade details and confirm the upgrade.

    The release package is now uploaded to the cluster and following upload, the upgrade begins. The Activities icon indicates the upload progress at the top of the page. When the upload is complete, the upgrade begins.

    You can monitor progress in the Activities page.

Upgrade from a pre-imported version

  1. Connect to the VAST Web UI.

  2. From the left navigation menu, select Infrastructure and then Clusters.

  3. Open the Actions menu  for the cluster and select Upgrade.

    The Upgrade Cluster dialog appears, with the currently installed build shown at the bottom left.

  4. Go to the Upgrade to Last Imported Build tab. If a package has been imported previously, the details of the package are displayed there.

  5. Click Upgrade.

  6. In Upgrade cluster to new version?, carefully set the options you need.

    Note

    If no options are selected, only the build is upgraded and the upgrade is non-disruptive.

    • OS Upgrade. Includes upgrade of VAST OS on the CNodes and DNodes as well as the core software build.

    • BMC Upgrade. Includes upgrade of BMC and all relevant components (FPGA, BIOS, UBM, CPLD and so on).

    • Isolate CPUs. Resets the configuration of isolated CPUs to an optimal configuration for the cluster according to a formula.

    • Force. Forces upgrade regardless of version or upgrade state. Use this option if you need to upgrade to the same version as already installed or if a previous upgrade failed without the upgrade state clearing.

      Caution

      Select this option only if directed explicitly by VAST Data Support.

    • FW Upgrade. For CERES, upgrades NIC firmware for BMC, MCU, PCI and Bluefield.

    • DR. Enables data reduction on a cluster that does not have data reduction enabled before upgrade.

    • Vms-only upgrade. Upgrades VMS version only.

  7. Click Yes, upgrade cluster.

  8. Review the upgrade details and confirm the upgrade.

    The upgrade begins. The Activities icon indicates the upload progress at the top of the page. When the upload is complete, the upgrade begins.

    You can monitor progress in the Activities page.

Upgrading via VAST CLI

  1. Connect to the management CNode using the local IP (not the VMS virtual IP).

  2. Transfer the package file to the management CNode, placing it under /vast/bundles. For example, if you are provided with a link to download the package from Azure:

    wget link -O /vast/bundles/filename

    (where link refers to the full download link, and filename refers to the name of the release package file.)

    Note

    It is important that the file is specifically copied to /vast/bundles and not to any other location.

  3. Run the following command to start VAST CLI:

    vcli -u <username> -p <password>
    
  4. From the VAST CLI run the cluster upgrade command.

    For example, for NDU of the software build alone:

    cluster upgrade --build-package /vast/bundles/release-<release number>-<pipeline number>.vast.tar.gz

    Alternatively, with VAST OS upgrade on the CNodes and DNodes:

    cluster upgrade --build-package /vast/bundles/release-<release number>-<pipeline number>.vast.tar.gz --os-upgrade
  5. Wait patiently for upgrade to complete.

    Initially, the build file is extracted and imported, which takes some time. Once that stage is complete, the upgrade begins and the progress is displayed in the CLI.