VAST Cluster Isolated Recovery With Commvault - Configuration Guide

Prev Next

Introduction

Backing up data is a no-brainer in today’s world, where businesses and lives depend on it. But what about restoring that backed-up data? I suspect that most businesses treat restoring backups like a jack and a spare tire in their car and know they have it but aren’t sure how to use it. Point restores focus to a particular file or, perhaps, to a previous state of a virtual machine, but ultimately, the ability to complete that task is not insurmountable. But now throw in a cyber-attack with the entire infrastructure potentially at risk – The layers of questions and uncertainty around the efficacy of a particular backup become an issue.

Most data protection vendors are familiar with the concept of a cyber vault, clean room, or isolated recovery environment where a secondary or even a tertiary infrastructure is set up and is completely separated from breached production and even primary backup storage and is used to scan, cleanse, and restore at least minimal viable company data. However, more questions arise around the complexity of the clean room, the time to build it, what can be automated, the cost, whether all of the very latest backup data is available, whether I have older recovery points available from weeks ago, and most importantly, how quickly required data and related workloads can be found, cleaned, and restored.

A lot of storage vendors go through the motions with solutions involving redundant arrays in both locations (for example, production & DR) and even a tertiary vault environment, but by that point the cost is already close to double or more.

Step into the VAST Data Platform and its unique disaggregated shared everything () design where the compute and capacity are connected over a high-speed NVMe fabric, with each being able to scale independently. This separation of compute and capacity gives the VAST platform the unique ability to isolate front-end CNodes into a dedicated network and environment, while the private back-end DNodes remain reachable from all CNodes.

The ability to physically separate front-end client connectivity from dedicated CNodes into a separate, isolated environment (minimum of 2) is the key to this solution.

Each CNode is dual-connected, with one connection going into the backend (DNode) NVMe fabric and then front-end connected to a customer’s clients and data mover servers, for example Commvault MediaAgent servers or HyperScale Flex nodes. Figure 1 highlights the physical layout of the environment. It shows the client side of two CNodes physically separated from the production client side into a separate isolated recovery environment (IRE) while the backend is still connected to the overall global namespace of the VAST cluster with multiple layers of data and storage resilience, provided by immutability, WORM, snapshots and ultra efficient erasure coding that has a Mean Time to Data Loss (MTDL) of over 2 million years.

Once this physical separation is complete, there are additional steps within the VAST cluster to craft the overall solution. These steps ensue.

Audience

This document is a high-level integration guide to assist administrators and others interested in understanding the workflow for configuring the VAST Data Platform in an isolated recovery environment (IRE) and/or cyber vault. The concepts will apply to any data protection/backup ISV, but this example highlights the value and cyber resilience brought to Commvault deployments.

For the most part, this paper will describe the basic solution and the specific VAST tasks needed to configure the isolated tenant environment for integration with any supporting backup software.

Solution

Diagram titled PRODUCTION showing VAST Data Platform isolated recovery solution with production and IRE sections, clients, NVMe fabric, cNodes and dNodes, and arrows indicating live sync and data validation

Figure 1 - VAST Data Platform Isolated Recovery Solution

Key Takeaways

  • Segregate CNodes (minimum 2) into an isolated network

    • Front-end CNodes are separated into a completely separate network.

    • Back-end DNodes are still connected to the private NVMe fabric and VAST’s global namespace, allowing separated CNodes to see all the data.

    • An isolated environment can be used to cleanse and restore data in record time.

  • VAST Global Namespace

    • VAST’s single global namespace allows the separated cNodes to still access all the data.

  • Superior Restore Capabilities

    • Restored systems and data do not consume additional space because they are deduplicated within the same system.

    • Data is local to the same VAST cluster so cleansing and restoring data are completed quickly.

Setup Overview

Using Figure 1 as a reference for a generic isolated room recovery, several layers and features need to be covered to understand how to build out an on-premises IRE. The topics include

Diagram showing two rounded workflow boxes side-by-side. Left box titled Production/Cluster Admin lists steps such as Configure snapshots on production buckets, Create Tenant, Create VIP Pool with sub-items and CommCell/Data Recovery. Right box titled IRE/Tenant Admin lists Tenant Preparation with bullet items (create identity policy, create user(s), create view policies, Commvault preparation) and CommCell/Data Recovery with recovery steps. Icons for buckets, briefcase, network, and recovery are shown. The diagram is captioned Figure 2 - IRE Workflow.

Figure 2 - IRE Workflow

Preparing the Environment

The following sections address the preparation needed to ensure that the production side of the VAST cluster is properly prepared for a crisis or cyber event as well as the isolated recovery environment (IRE). The IRE preparation could be completed at any time, but it’s recommended to complete it along with the production setup so that rehearsals, including the option to perform threat scanning (ISV capability), can be performed to ensure an end-to-end cyber recovery and forensics process is well understood, practiced, and completed without any hiccups.

Production Prep

The most important aspect of being prepared in the production environment for a cyber-attack is ensuring that the VAST views are appropriately snapped, as this will be the critical component used during recovery. In this example, user data, along with the Commvault data plane views, will be on a snapshot schedule, so they can both be recovered in the IRE.

Scheduling Snapshots

Data Protection, in the context of a VAST Data cluster UI, focuses on snapshots and replication. In this solution, a snapshot schedule (policy) needs to be created and assigned to the appropriate VAST views to ensure sufficient time points for recovery. The following steps outline the procedure for creating a snapshot and apply to any number of views.

To start, in the auto-hide left windowpane, select Data Protection and then Protection Policies. Once on the Protection Policies page, select Create Protection Policy. This brings up the Add Protection Policy wizard shown in Figure 3, which is already fully populated. Craft the snapshot policy according to the business’s RPO needs.

Add Protection Policy

Figure 3 - Creating Snapshot Policy

After adding the appropriate snapshot schedule, simply click Create. The policy will now show as in Figure 4. In this example, the two highlighted policies were created and cover different snapshot frequencies for the data and the CommCell server.

Data Protection list

Figure 4 - Protection Policy Created

With the policy created, it can now be applied to appropriate views. To do this, go back to the left pane window and select Data Protection and then Protected Paths. This opens the window shown in Figure 5. Fill out the fields, entering the name and the path to be protected. In this example, the name matches the path for easy reference, but it can be named according to any convention. Finally, select the protection policy that was created previously and click Create.

Create Local Protected Path modal dialog showing a dark blue form with fields labeled Tenant, Name, Path, Protection policy, and action buttons Dismiss and Create

Figure 5 - Creating Protected Path

Figure 6 highlights the protected paths that have been created successfully.

Data Protection table interface showing a list of protected paths with columns such as Name, Role, State, Health, Local Path, Tenant, Protection Policy and a red highlighted row for /backups/commvault/buckets/critical-data

Figure 6 - Protected Path Created

The process just highlighted will be applied to the major components of this solution – the CommCell DR location and a set of files. Both of these will be restored in the isolated recovery environment. But first, there are steps to set up the IRE. Those are addressed next.

Creating the Isolated Recovery Room

This section is where the physically separated CNodes are separated by the VAST Operating System, forming the isolated room. There are two major components to this: creating a tenant and a VIP pool.

Note: The settings and configuration discussed for multi-tenancy are a limited subset of the full functionality (For more information, see VAST Documentation).

Tenant Creation

The VAST Data Platform has extensive multi-tenancy support, and creating a separate tenant will be used to enhance and harden the CNodes’ physical isolation. Having the CNodes and VIP Pool assigned and segregated into their own tenant adds another layer of security by isolating data paths for serving different (IRE) clients and data movers. All read and recovery actions will occur inside the new vault or IRE tenant.

To create a new tenant, go to the left windowpane of the VAST UI, select Element Store, and click on Tenants (Figure 7). In the upper right of the Element Store screen, click on Create Tenant (Not shown). In the new tenant wizard (Figure 8), enter an appropriate name and an optional domain (this overrides the tenant name used in the URL management link for this tenant).

Left navigation menu of the VAST UI showing the Element Store section expanded with the Tenants item highlighted in a red box; screenshot of the VAST UI left windowpane menu.

Figure 7 - Select Tenants

Screenshot of Create New Tenant UI — left navigation showing General* and Providers And Users Access. Main panel titled General with Tenant name * field set to T3-IRE and a highlighted Login URL Preview: https://10.143.15.203/#/login/t3-ire.

Figure 8 - Configuring New IRE Tenant - Name and Provider

The login URL of this new tenant is critical to being able to manage the IRE environment. Make sure to note this down in a safe location and/or bookmark in a web browser.

Screenshot of Providers And Users Access UI — main panel titled Providers And Users Access. A table/list shows VAST Provider (t3-ire) and other provider options; the t3-ire option is highlighted with a red circular annotation and a radio selection indicating it is chosen. Left navigation shows tenant creation steps.

Figure 9 - Selecting Local VAST Provider

In the Providers and Users Access section (FIGURE 9), there are 2 options for allowing users to access configured views within the VAST cluster. The first is the VAST Provider, which is a collection of local users configured on the cluster. The second is Active Directory. Both can be configured at the same time; however, the base assumption in the recovery environment is that initially there might not be any Active Directory for user validation. So, the tenant will be configured with a VAST local provider where the creation of a new local user will be used for S3 bucket authentication.

For ease of configuration, the new provider can be created during tenant creation. To do this, select VAST Provider and then click +Add New to create a new local provider (FIGURE 9). The Create Local Provider wizard pops up (FIGURE 10). Enter an appropriate name for the provider. Each environment design will be different, but to limit the surface area of what the new tenant can do, only tenant admin capabilities were assigned. (A local user will be created once logged into the tenant.) Click Create when finished and then select the newly created provider as shown back in Figure 9.

Create Local Provider modal dialog titled Create Local Provider showing form fields Name (value t3-ire) and Description (Local VAST Cluster Users for IRE), checkboxes Cluster admin (unchecked) and Tenant admin (checked), and action buttons Dismiss and Create.

Figure 10 - Creating New VAST Provider for Tenant

In the User Access Management section of the Create New Tenant wizard, a pre-configured user will need to be created for login access to this tenant and since it can’t be added during tenant creation (Figure 11) it’ll be addressed in the next section.

Create New Tenant screen showing the left navigation and the User Access Management panel for granting tenant admin users later.

Figure 11 - Tenant Access Granted Later

Moving further down the tenant configuration wizard is the IP Addresses for Client Data Access section. The basic concept of access to the VAST cluster is that all I/O goes through the secure IP addresses of a VIP Pool, which are assigned to specific CNodes. The tenant concept adds another security layer to this, in which a client would now need access to the tenant first through IP Addresses for Client Data Access. Access to the tenant grants access to the CNodes, which in turn grants access to the VIP Pool.

The IP Addresses entered here in this section are given permission to access this tenant. Any addresses from clients outside of the range(s) will not be able to access this tenant. This is an additional security layer that can be used to better insulate systems within the recovery room from the production environment. In this example, the Commvault CommCell, the media agents, and an example client will be allowed to access VAST through this tenant. The configuration is straightforward by simply entering a start and end IP address for the range and then clicking Add to Table. Here, twenty-two IP addresses are added to the table (Figure 12).

Dark-themed UI screenshot titled IP Addresses for Client Data Access showing Start IP and End IP fields (172.29.6.79 and 172.29.6.100) and a table row with Start, End and Number of IPs columns; red box and red arrow annotations highlight the End IP and the table row.

Figure 12 - Creating Tenant IP Address Client Access

The last important piece to consider during tenant creation is the default view policies (Figure 13). These policies are nice to have when configuring views for quick verification. It is generally considered a best practice to configure separate, more targeted view policies to cater to specific view requirements. That said, creating and having the default view policies is handy and not discouraged.

Dark-themed UI screenshot titled Advanced Protocol Settings showing a panel with Create default view policies checkbox selected and related common settings on the right side.

Figure 13 - Allowing Default Policies (optional)

Dark-themed UI screenshot of the main Element Store Tenants table showing a tenant row highlighted in red with checkbox, tenant name T3-IRE, client IP ranges column displaying 172.29.6.[79-100

Figure 14 - New Tenant Created

There are additional settings in the wizard that are not necessary to address here; simply click Create to complete the new tenant. This new tenant is highlighted in Figure 14.

Tenant User Creation

To be able to login into the new tenant a user needs to be created. The following example highlights how to create a tenant user with administrative-level controls. The user permissions can be configured as needed.

Role Creation

The first step is to ensure that a proper role is created for the user. The role defines the set of permissions the user will have and is used when creating the tenant manager’s account. To create a role, go to the left panel, select Administrators, then Roles. In the upper right corner, click on Create Role. This will bring up the Create Role wizard (Figure 15).

Create Role modal showing Tenant Admin user type selected, tenant dropdown (T3-IRE), role name field, and a permissions table with a Realm checkbox and columns for Create, View, Edit, Delete — dark teal themed UI screenshot with red outline highlights

Figure 15 - Creating a Role

Select Tenant Admin as the user type, select the tenant (T3-IRE) that was created previously, and then give it an appropriate name describing that it is a role. The default realm was used to grant the role its permissions set, since the goal was to grant it admin capabilities from a tenant perspective. The Realm check box was clicked to quickly assign all permissions (Figure 15). When finished, click Create.

Administrators  Roles table showing a highlighted row for T3-Tenant-Role with tenant column T3-IRE — dark teal dashboard table screenshot with red bordered selection

Figure 16 - Tenant Role Created

Manager Creation

With the role created, it will now be assigned to a tenant manager. To create a manager, go to the left panel, click Administrators, then Managers. In the upper right corner, click on Create Manager. In the wizard (Figure 17) fill out the form with the appropriate user information and then ensure the User type is set to Tenant Admin and then select the tenant as well as the role that was created previously. When finished, click Create.

Create Manager wizard UI on a dark teal background showing form fields — Username t3, First Name Isolated, Last Name Recovery, password fields and criteria checklist, user type radio options with Tenant Admin selected, and Roles section highlighted

Figure 17 - Create Manager Wizard

The completed tenant manager will be similar to what's shown in Figure 18. This now completes the tenant configuration.

Manager list UI showing a row highlighted for user t3 with columns First Name Isolated, Last Name Recovery, Roles T3-Tenant-Role, Tenant T3-IRE, Default No, Active Yes

Figure 18 - Manager Created

Virtual IP (VIP) Pool Creation

On a VAST cluster, all client-side access is gained through Virtual IP (VIP) Pools. A VIP Pool is an IP range created for clients to access VAST views (NFS, SMB, S3) either directly via IP addresses or through a DNS domain name. Using a DNS name allows for automated distribution of the IP range, improving IO spread across all the CNodes.

A VIP Pool can be assigned to all CNodes in a cluster or to a specified set of CNodes; at a minimum, two CNodes are required in any VIP Pool. To ensure that no other clients have access to the isolated CNodes, only a single VIP pool should be assigned to them.

This functionality is how a vault or isolated recovery environment will be created and separated — both from hardware and software. We'll create a VIP pool and assign it, and only it, to the isolated CNodes.

To start with the VIP pool creation — in the left windowpane of the VAST UI, select Network Access, then select Virtual IP Pools (FIGURE 19).

Screenshot of the VAST application UI showing the left navigation menu with Network Access expanded and the Virtual IP Pools menu item highlighted; performance widgets (Bandwidth, IOPS, Latency) visible on the right

Figure 19 - Accessing Virtual IP Pools

Now, click the button in the upper right corner – Create Virtual IP Pool (Figure 20). This will bring up an appropriate wizard (Figure 21).

Screenshot of the Virtual IP Pool Details table in the VAST UI showing multiple VIP pool rows (Name, IP Ranges, Subnet-CIDR, VLAN, Vast DNS Domain Name, CNodes, Tenant) with the Isolated Recovery row outlined/highlighted

Figure 20 - Virtual IP Pool Details

Starting with the tenant pull-down menu, select the Isolated Recovery tenant created in the previous step, then give the VIP Pool a name. This ties the VIP Pool to that tenant, ensuring that only this tenant has access to the VIP Pool. In the Resource Selection area, uncheck Include all CNodes. This now gives the user the ability to select specific

CNodes to be used for this VIP Pool.  As shown in Figure 21, at least 2 CNodes have been selected.

Note: At a minimum, 2 CNodes must be assigned to a VIP pool.

Dark-themed Isolated Recovery UI screenshot showing General and Resource Selection panels. Left panel lists All possible CNodes with multiple cnode entries and an Include all CNodes checkbox; right panel shows Selected CNodes (2) with two cnode items. Red highlight boxes and an arrow indicate the selection flow from the left list to the right list.

Figure 21 - Create VIP Pool - Isolate CNodes

Moving to the IP Range List, this will define the number of IPs in the VIP Pool.  The best practice is to have at least one IP per CNode, so that each IP is assigned to a single CNode.  With a minimum 2 CNode configuration, 2 IPs are required.

Dark-themed IP Range List and DNS Configurations UI screenshot showing a highlighted Start IP / End IP table row with values 172.29.93.1 and 172.29.93.8, and a DNS Configurations box showing Virtual IP Pool Domain name: ire and DNS Service FQDN: ire.tmphx203.vastdata.lab. Red outline boxes emphasize the IP row and DNS fields.

Figure 22 - Create VIP Pool - IPs and DNS

The last thing to consider in creating the VIP pool is its DNS Configuration. Again, the assumption in the event of a cyber recovery is that most services may not be available initially. Preconfiguring the VIP pool with a DNS entry can be useful if part of the recovery process is to have the DNS service working as one of the early steps. Also, configuring the system’s ‘hosts’ file with this DNS entry is a potential workaround for ensuring this functionality is available before the DNS system is active.

Once finished with these settings, click Create. The finished VIP Pool is shown in Figure 23 with the IP range, DNS entry, selected CNodes, and tenant assignment.

VAST Network Access screen showing the Virtual IP Pools table. The table row labeled Isolated Recovery is visible with columns for IP Ranges (e.g., 172.29.93.[1-8

Figure 23 - VIP Pool Created

Now both pieces, the tenant and VIP Pool, have been created, and the vault or isolated recovery environment infrastructure is set. A few things need to be completed inside the IRE environment; log in as the tenant admin.

Preparing the Isolated Recovery Environment

Now that the IRE tenant has been set up, there are a few things to configure in advance within it. A local user needs to be created with keys and an identity policy. Also, a default S3 policy will be needed, with the local user assigned to it and granted list bucket permissions.

Tenant Login

Enter the specific login URL generated during tenant creation (Figure 8) in a browser. Then, using the previously created tenant administrator, log in to the tenant (Figure 24). Notice the tenant is listed below the VAST logo to remind the user which tenant they’re logging into.

VAST login screen showing the VAST logo with tenant label t3-ire highlighted; a login card displays fields with Username t3 and a masked Password entry and a Login button.

Figure 24 - Logging Into Tenant

Create Identity Policy

Identity policies are a specific list of S3 API permissions granted to users. For proper interoperability, Commvault has a specific set that should already be in production and can be copied to the recovery tenant for use. In the recovery tenant, in the collapsible left column, select User Management, then Identity Policies.  In the upper right, click the Create Policy button.  That brings up the Add Policy wizard (Figure 25).  Give the policy a name, then copy and paste the production-side JSON text into the window on the right.  Then click Create.

Add Policy wizard screenshot showing a dark-themed policy form on the left (General Policy Details, Define policy fields) and a code editor on the right containing production-side JSON policy text highlighted in a red box.

Figure 25 - Creating Identity Policy in IRE

This policy will now be applied to the new local tenant user that’s created next.

Create Local User

Within the IRE tenant, it’s assumed that Active Directory may not be available during a cyber recovery.  When the tenant was set up, a local provider group was created specifically for this tenant.  Now we’ll populate it with a new user.

The User tab is located right next to Identity Policies. Alternatively, click the left column, select User Management, then Users.  In the upper right corner, click on Create Local User (not shown), and the Add User wizard will open (Figure 26).

Add User wizard screenshot showing dark-themed form with General fields (Name, UID), Leading group and Groups dropdowns, an informational note about S3 Access keys, and toggles for Allow Create Bucket and Allow Delete Bucket.

Figure 26 - Adding New User in IRE

To create the user, simply give it a name, and any random UID will suffice.  This user will only need read permissions to restore data, but it may also need additional permissions, so grant the appropriate bucket permissions.

Lastly, ensure the newly created identity policy is assigned to this user, then click Create.

Create View Policy

Whether or not default view policies were created during the tenant creation, it's a good idea to separate specific use cases into their own view policy. To create a new view policy, go to the left column and select Element Store and then View Policies. In the upper right corner, click on Create Policy.

Screenshot of the Add Policy wizard showing the General section with Name set to ire-s3 and Security flavor set to S3 Native, with the Name and Security flavor fields highlighted in a red box

Figure 27 - Creating IRE S3 View Policy

This brings up the Add Policy wizard (Figure 27). Give the policy a name and set the security flavor to S3 Native. Pinning the policy to a specific Virtual IP pool is unnecessary, as there is only a single VIP pool. Even when creating an S3-only view policy, the NFS group membership source needs to be set. Keeping this environment as secure as possible, select Providers (Figure 28).

Screenshot of the Add Policy wizard showing the NFS section with the Providers dropdown open and Providers selected/highlighted

Figure 28 - Setting NFS Group Membership

Scrolling further down in the wizard, in the S3 section, ensure that the user created previously is added to the Bucket list permission for users (Figure 29). When finished, click Create.

Screenshot of the S3 section of the Add Policy wizard showing the Bucket listing permission (users) field with a user (example tag t3-cvlt) added and highlighted

Figure 29 - Adding User to Bucket Listing of Policy

When finished, the view policy should look like what is shown in Figure 30.

Dark teal dashboard screenshot titled Element Store showing a table of view policies. One row labeled ire-S3 is outlined in red; columns visible include Name, Flavor, NFS Read Write, NFS Read Only, and Creation Time with timestamps.

Figure 30 - IRE View Policy Created

This completes all the IRE preparatory work, and a cyber recovery simulation can now be performed.

Cyber Recovery

In the event of a cyber-attack, the recovery process is quite simple, but as anyone knows, having a spare tire and a jack is not the same as being able to do it successfully. Going through a cyber recovery should be rehearsed and documented.

With all the IRE prep work completed, there is only one task on the production side (default tenant): locating an appropriate recovery point and cloning that snapshot. If, after this process, it is determined that the recovery point did not contain the required data, that clone can be deleted and the process repeated on another point-in-time snapshot.

Clone Recovery Point

Cloning the desired snapshot recovery point is the only task that needs to be performed in the production environment during a recovery. It can be performed as many times as necessary on as many snapshots as necessary to locate an appropriate snapshot with the desired state of the data.

Screenshot of a Data Protection UI — dark teal header with tabs (Protected Paths, Protection Policies, Replication Peers, S3 Replication Peers, Restore Points, Snapshots, Global Snapshot Clones). Main area shows a table of snapshot entries with columns like Name, Path, Policy, Type, Indestructible, Creation Time. A context menu is open on a snapshot with options such as View, Edit, Map Snapshot Volumes, Clone (highlighted), Duplicate And Edit, Delete, and Create New. The UI has checkboxes for rows and a Filters bar with Name Contains: dr.

Figure 31 - Creating Clone of Desired Snapshot

To locate the desired snapshot, go to the left panel, select Data Protection, then Snapshots. As shown in Figure 31, simply right-click the snapshot and select Clone.

Modal dialog titled Add Global Snapshot Clone showing form fields — Name: Commcell-DR-2025-09-03-22-22; Target tenant: T3-IRE; Target Path: /commvault/clone/cmdr/; a Background sync toggle on the right; and action buttons Dismiss and Create at the bottom.

Figure 32 - Clone Details Including Tenant

In the clone wizard that appears (Figure 32), enter an appropriate name and for the target tenant select the IRE or vault tenant that was created previously. This will make the clone available only in the IRE or vault tenant. To ensure only data that is needed is read from blocks in the original snapshot (i.e. a thin clone), and not a full physical copy of underlying data blocks, toggle the Background sync off. Finally, enter the path the clone will use in the IRE, and make sure to note it. Once logged into the tenant, this path will not be visible until a view is created using it. Click Create when finished.

Data Protection UI showing the Global Snapshot Clones tab with a table of clones. The highlighted row shows Commcell-DR-2025-09-03-22-22, tenant T3-IRE, target path /commvault/clone/cmdr/, source snapshot and sync progress columns — illustrating the created snapshot clone entry.

Figure 33 - Snapshot Clone Created

Switching to the Global Snapshot Clones tab, the created clone can be seen (Figure 33). Repeat this same process for the CommCell data plane view and any other views that need to be recovered.

Creating IRE Views

With the appropriate snapshots cloned into the IRE tenant, now it’s possible to create the views, specifically S3 buckets, from those clones and then present them out for recovery actions. To begin creating the views, log in to the VAST UI using the tenant URL created during tenant creation.

On the left panel, click on Element Store → Views and finally click on Create View in the upper right. This brings up the Add View wizard (Figure 34). The first entry is the path that was created in the previous section when the snapshots were cloned. Enter in the S3 view policy that was created in the IRE. Select S3 Bucket as the protocol and give it a bucket name.

Dark-themed Add View wizard screenshot showing the General panel with Path field (/commvault/clone/cridata), Create directory checkbox, Policy name (ire-S3), Protocols set to S3 Bucket, and an S3 bucket name field; red curved arrow annotations point between fields.

Figure 34 - Creating IRE Views

Larger dark-themed Add View wizard screenshot showing S3 settings: Bucket Owner selection (tenant user highlighted), Enable Object lock toggle enabled, Versioning and Object Lock sections visible, and large red circular arrow annotations indicating settings to configure.

Figure 35 - Creating IRE Views - Bucket Owner/Worm

The created S3 bucket views are shown in Figure 36. Again, in this example, the CommCell database will be recovered along with some user data.

Screenshot of the VAST Admin Element Store UI showing a dark teal table view with filters and annotated red number callouts; captioned figure image centered on the page.

Figure 36 - Finished IRE Views

Recover CommCell Server

The first step in recovering the environment is to restore the CommCell server so it has the control plane metadata needed to enable restores from the backups in the S3 bucket from the cloned global snapshot (from the last step).

With the CommCell database backed up in an S3 bucket, the CommCell server can be recovered quite easily. There are a number of different methods for doing this recovery, but this example focuses on ingesting the backed up CommCell database into a fresh install of Commvault since the production version and any snapshot of a production virtual machine could also be compromised, and building from a fresh copy is the safest and most expedient approach to recovery.

Assuming a fresh, matching version of Commvault has already been installed in the IRE, here is one method for restoring the database (See the Commvault documentation for additional methods).

Restoring CommServe Database

When importing a backup copy of the CommCell database, the new Commvault server version must, at a minimum, match the production version exactly. That means the major, minor, and even maintenance pack releases have to be identical or later. A good practice to keep this available without needing internet access (which may be blocked or compromised) is to have the install bits already downloaded and handy. Furthermore, whenever the production server is updated, install a fresh version in the IRE and keep it ready.

To start, we’ll grab the backed-up database files from the bucket created in the IRE using the cloned snapshot. There are a couple of ways to grab these files. Commvault’s installation includes its own S3 utility, CloudTestTool.exe, which is a text UI. It’s located in the default installation directory, typically:

<install directory>\Commvault\ContentStore\Base

Starting the program opens the text UI shown in Figure 37. To connect to the bucket, configure the tool with the appropriate user credentials and endpoint. To start, simply select option 3 for S3-compatible storage.

A black terminal window screenshot of Commvault's CloudTestTool showing a Select Vendor Type: menu with a numbered list of cloud storage vendors (e.g., S3 Compatible Storage, Amazon S3, Microsoft Azure Storage, Google Cloud Storage, etc.) and a command prompt displaying Select [1

Figure 37 – Commvault’s CloudTestTool

From there, enter an IP address that’s in the tenant’s VIP Pool and add the S3 keys and bucket name (Figure 38). Here is where knowledge of the bucket contents might become an issue. Unless the exact directory within the bucket is known, recovery may require a different approach (discussed next).

A black terminal window screenshot showing CloudTestTool prompts and entered values for configuring S3: Enter Host Name, Enter Access Key ID, Enter Secret Access Key, Bucket, and a final prompt Press '1' to continue or '0' to back to the previous menu [1

Figure 38 - Configuring S3 Endpoint and Credentials

Assuming the exact source path is known, the next prompt asks for that source location and where to download the contents (Figure 39). In this example, the downloaded backup copy of the Commvault database is saved to the desktop and will be imported into Commvault.

There is another approach that is slightly simpler because it uses a UI-based S3 utility. This example uses S3 Browser, which is a free download and intuitive to use.

A black command-line window titled CloudTestTool showing a menu of operations and a download progress message.

Figure 39 - Entering Source/Target Location and Downloading

Just as in the text UI, this tool also requires the endpoint and S3 keys to be applied within the new account wizard (Figure 40).

S3 Browser Edit Account dialog showing fields such as Display name, Account type (S3 Compatible Storage), REST Endpoint, Access Key ID, and Secret Access Key.

Figure 40 - S3 Account in S3 Browser

Once configured, browsing the bucket's contents is as easy as using a file browser. Within the bucket is the original directory containing all the backed-up versions of the Commvault database (Figure 41). In this example, the latest backup is being downloaded to the desktop.

S3 Browser application window showing a left bucket panel with 'cmdr', a path field showing '/ DR/', a main pane listing folders like SET_313/ SET_312/ etc., and a right-click context menu with the 'Download' option highlighted

Figure 41 - S3 Browser for UI-based downloading

Now that there’s a backup copy of the Commvault database on the desktop, it can be easily imported into the fresh Commvault installation. Starting from the screen in Figure 42, click on the Next button when ready.

CommServe Install Window dialog titled 'Commvault Client Computer Information' showing fields like Display Name, Host Name, IP Address(es) and a 'Next' button at the bottom right

Figure 42 – CommServe Install Window

On the next window (Figure 43) select the Use Existing Database radio button and check the CommServ box as this is an existing CommServ database. Then browse and select the location of the downloaded database files.

Commvault Database Install Option dialog showing Use Existing Database selected; CommServ checkbox checked; other database checkboxes (AppStudioDB, CVCloud, DM2, HistoryDB, TaskManagerDB, WFEngine) visible; Database Dump Location field set to C:\Users\Administrator\Desktop\SET_313; Back and Next buttons visible

Figure 43 -Backup Database Directory Selected

Commvault Database Restore Options dialog showing radio options Staging / Troubleshooting and Recovery / Production with Recovery / Production selected; descriptive text about migrating or recovering a CommServe; Back and Next buttons visible

Figure 44 - Selecting Restore Option

When ready, click the Next button, then select Recovery/Production. Commvault will then begin importing the database. This process can take varying amounts of time depending on the hardware used.

Once the CommServ is recovered, production data and VMs can be restored. For VMs, the cloned snapshot could be presented out as an NFS datastore and directly attached to an ESXi cluster in the IRE. For file data, shown in the next section, that will be presented back out as a bucket, the same way it was backed up.

Commvault IRE Configuration

With CommServ restored, there are a couple of tasks to perform to build out and reconnect to all retained backup data in the restored bucket. In Figure 45 at the top, two new servers are seen. One is a MediaAgent for data movement, and the other is a client that will later be used to highlight data restoration.

In addition, the Commserv (cvm) can be seen as it was restored from the previous section. The old media agents are also shown, but with an unknown status, since they are on a different network.

CommCell Servers UI screenshot showing a list of servers and MediaAgent entries with left-hand navigation menu

Figure 45 – New MediaAgent and Client Added

Adding Tenant User

For the CommServ and MediaAgent servers to access recovered data from an S3 bucket, the user created in the VAST IRE tenant must be added to the CommServ server with the corresponding access and secret keys. To do this, under Manage, click on Security, then select Credential Vault (Figure 46).

Security dashboard screenshot showing tiles such as Users, User groups, Roles, Key management servers, and Credential vault

Figure 46 – Select Credential Vault to Add Tenant User

In the next window, a list of current users will be shown (Figure 47).

Credential management table screenshot showing credential names and types with the Add button highlighted

Figure 47 - CommCell Users and Credentials

Click Add in the upper right to bring up the Add Credential wizard and then fill in the appropriate information — choosing Cloud Account and the VAST Data Platform from the pull-down menus (Figure 48).

Add credential dialog showing fields: Account type set to Cloud Account, Vendor type set to VAST Data Platform, Credential name t3-cvlt, Access key ID value visible, Secret access key masked, Description field, and Cancel / Save buttons at the bottom.

Figure 48 - Adding Tenant User

The new user is now visible in the list, as shown in Figure 49.

Manage credentials list showing multiple credential entries with one row highlighted (credential name t3-cvlt and type VAST Data Platform) indicating the newly added tenant user.

Figure 49 - Tenant User Added

Modifying Mount Path

The production-side MediaAgents that had access to the buckets did so through a production-side VIP Pool. The IRE has access to the global cloned snapshot data and buckets via an IRE VIP Pool created in a previous step. Now, to access the recovered data in the IRE, a simple modification to the Commvault library mount paths is all that’s needed to connect a new IRE MediaAgent server(s) to the new bucket.

To do this, click on Storage and then Cloud (FIGURE 50). Since this example highlights file data recovery, the CriticalData library in the figure is selected for modification.

Commvault Cloud UI screenshot showing a Cloud libraries list table with columns Name, Vendor type, Status, Storage class, Region, and Used space; left navigation menu visible.

Figure 50 – Select Library to Modify Existing Path

Within the library configuration in the Backup Locations tab, are the old, assigned media agents, and at the bottom, the link for the bucket (Figure 51). Click on that bucket link at the bottom.

Commvault Cloud UI screenshot showing the selected library bucket view with Associated MediaAgents table and the Bucket section listing the bucket entry at the bottom.

Figure 51 – Select Library Bucket to Modify Paths

This now opens a General window containing information about that bucket (Figure 52). There are several items to address here. First, toggling on “Disable backup location for future backups” is a good best practice so that backups don’t begin unexpectedly and resources aren’t used for that purpose.

Cloud management UI showing [cv-media61

Figure 52 - Select Bucket Link

Second, the new MediaAgent server that was added previously needs to be added and connected to the bucket so it can take over data movement. To do that, click on Add MediaAgent. This opens the wizard, where the MediaAgent can be added (Figure 53).

Note: Another best practice to consider is deleting all the old media agents to tidy up the environment, introducing only new media agents and the recovered CommServ.

Add MediaAgent dialog window showing a list of MediaAgent names with checkboxes and a selected row cv-ire-ma93w; dialog title Add MediaAgent.

Figure 53 - Adding New MediaAgent

The last task is to modify the access path that the new MediaAgent(s) will connect to the bucket. To modify the access path, click the Bucket link corresponding to the new MediaAgent.

Note: In this environment if an access path is changed then all connected MediaAgents will change however in some environments a Commvault setting may be enabled that allows for each media agent to have its own unique path.

This now brings up the Edit Cloud Access Path window (Figure 54). Modify the Service Host setting accordingly. Initially, without any DNS, it may be necessary to only use a single IP address until additional services are brought online, at which point an FQDN for the VAST VIP pool can be used.

Edit cloud access path dialog showing fields — Type: S3 Compatible Storage; Name: CriticalData-Library; MediaAgent: cv-ire-ma93w; Service host set to http://172.29.93.1; Credentials and Bucket fields visible

Figure 54 - Modify Service Host to IP or VIP Pool in IRE

After finishing, the modified access path will show as in Figure 55.

Cloud access paths table showing an entry for media agent cv-ire-ma93w with Bucket 'critdata' and User name highlighted as http://172.29.93.1/t3-cvlt; columns include MediaAgent, Bucket, User name, Access, Accessible, Actions

Figure 55 – Modified Cloud Access Path

Restoring Data

With the CommCell environment now able to access the recovered data through a new S3 bucket, it's time to demonstrate data recovery. Since this is a recovered Commserv server, all of the job history for all of the original retention that was in production is still intact.

Click on Jobs, then the Job History tab, and, with or without filters, peruse the job history to pinpoint an appropriate backup. Then, click the three-dot ellipsis for the backup, and select Restore (Figure 56).

Application dashboard showing a list of backup jobs with columns for Job ID, Operation, Server, Agent type, Subclient, Storage pool, Size, End, Elapsed, Retain until, Status, Error description, Error code, and an Actions menu. Left sidebar navigation is visible and a small context menu is open near the bottom-right of the screenshot.

Figure 56 – Select Restore on Backup

This opens a file directory to search for specific files for backup, or clicking at the root level would perform a full restore (Figure 57).

Backup content screen showing a file directory tree on the left and a table of files on the right with columns for Name, Size, Modified time, and Backup time. The interface includes toolbar actions like Restore and Download and the left navigation bar is visible.

Figure 57 - Select Data to Recover

After selecting the files, complete the restore wizard by filling in the additional restore options, then click Restore (Figure 58).

Screenshot of a Restore options dialog showing Destination field, checkboxes for ACLs and Data (Data checked), Restore to original folder option, Destination path C:\Users\Administrator\Desktop, Number of streams set to 10, options including Unconditionally overwrite if it already exists, an Impersonate user toggle, a When the job completes, notify me via email checkbox, and footer buttons EQUIVALENT API, CANCEL, and RESTORE.

Figure 58 - Select Restore Location

Summary

Traditional air-gapped and clean-room recovery solutions typically require building and maintaining a completely separate storage infrastructure in your data center or COLO, and typically at a secondary (DR) and/or tertiary location, driving up cost, complexity, and recovery time during a cyber event. The VAST Data Platform eliminates this requirement through its disaggregated shared-everything (DASE) architecture, which decouples compute and capacity over an NVMe fabric. By isolating front-end, client-facing CNodes into a separate network while maintaining secure access to the shared backend DNodes and global namespace, VAST enables an isolated recovery environment and/or Vault within the same cluster. This approach allows organizations to rapidly identify clean data, perform threat analysis and cleansing, and restore operations without duplicating storage infrastructure, while having all of the backup history (short- and longer term) available in just minutes in an immutable form when WORM storage (object) locking is configured in conjunction with the DP backup software.

Along with multi-tenancy and the ability to clone and present data through multiple protocols into the IRE, this approach not only enables recovery of file data, VMs, and databases, but also of the CommServ server.