VAST Cluster 5.0.0-SP24 Release Notes

Prev Next

Upgrade to this version is supported from VAST Cluster 5.0.0 up to 5.0.0-SP19 and from VAST Cluster 4.7.0 up to 4.7.0-SP28. No upgrade from any previous version.

Note that a direct upgrade may not be supported from hotfix builds. Consult VAST Support regarding the upgrade if you may be running a hotfix build.

To obtain the download package for VAST Cluster 5.0.0-SP24, reach out to your VAST Customer Success Engineer.

Enhancements in 5.0.0-SP24

Install & Upgrade

  • ORION-164487: This update prohibits VAST OS upgrades to versions greater than 12.9.x. for carrier-based Sanmina DBoxes (those providing dual-port access to single-port drives, i.e. Active-Active).

  • ORION-134629: Added the Bond and Northband options to the list of options available for selection in the Management network field of the VAST Easy Install wizard.

Networking

  • ORION-164916: Added a capability to fine-tune the amount of log messages written by VAST DNS. Suppressing DNS logging may help eliminate the dropping of DNS requests in case of a very high load on the DNS server.

Quotas

  • ORION-167022: Updated quota routines to respond with HTTP status code 507 (Insufficient Storage) if the quota limit was reached when attempting to write to an S3 bucket. Prior to this change, HTTP status code 500 was returned (Internal Server Error).

Protocols

  • ORION-154258: This update ensures that a file being created on an S3 and NFS-enabled view controlled with the S3 Native security flavor gets the same ACL regardless of the protocol (S3 or NFS) used to create it. Prior to this change, the ACL set by an S3 client could differ from the ACL set when creating the same file via NFSv3 or NFSv4.1.

SMB

  • ORION-154019: Added a capability to disable listing of snapshots (.snapshot) when listing the contents of a directory.

Protocol Audits

  • ORION-141718: Enhanced audit logging of changes in network settings to make the audit log entries consistent with entries for other similar update events.

Replication

  • ORION-168407: Added the ability to deploy S3 replication and asynchronous replication on the same protected path. To do so:

    1. Set up S3 replication in VAST Web UI.

    2. Create a replication stream for async replication in VAST CLI, for example:

      vcli: admin> replicationstream create  --protection-policy-id 7 --name "Profiles share"
        --source-dir /profiles   --target-dir /profiles2   --tenant-id 7 --remote-tenant-g
        uid df3324dc-d6a0-469e-8d57-d29ccb098b9a
  • ORION-168404: Updated the logic behind protected paths used for S3 replication so that:

    • The user is not required to enter a path on the remote peer for S3 protected paths.

    • When displayed in VAST CLI, only fields relevant for an S3 protected path are shown (as opposed to async replication protected path).

  • ORION-149378, ORION-144187: Added validations to ensure that a path exposing an S3 bucket does not contain any subdirectories on which protected paths are defined, and also that the source directory of a protected path does not include S3 bucket paths.

    The new validations apply when creating or updating a view or a protected path.

    After an upgrade to VAST Cluster 5.0.0-SP24, VMS raises an alarm for each offending view or protected path that already exists in the cluster.

Authentication & Authorization

  • VAST Cluster now supports Kerberos authentication for NFSv4.1 over RDMA connections.

VMS

  • ORION-161204: The following event definitions and alarms are now reported for non-root users:

    • SSD <ssd> serial: <serial> link speed low link speed detected

    • SSD <ssd> serial: <serial> low link width detected from DNode {1|2}

    • SSD <ssd> critical warning <warning>. warning code is <code>

  • ORION-69064: VAST Cluster analytics now includes bandwidth, IOPS and latency metrics per virtual IP pool.

VAST Web UI

  • ORION-102759: Added the Duplicate and Edit action to the actions menu for protection policies and protected paths (Data Protection -> Protection Policies or Protected Paths -> right-click a policy or a path).

  • ORION-101982: The CNodes page (Infrastructure -> CNodes) now features the Leader column that indicates whether the CNode is the leading node.

  • ORION-95148: Added the Duplicate and Edit action to the actions menu for a support bundle in the Support -> Support Bundle page.

VAST Prometheus Exporter

  • ORION-150877: The following metrics can now be collected through VAST Prometheus Exporter:

    • window_runtime__avg

    • queue_latency_ns_fiber__avg

    These metrics are used to monitor VAST CPU utilization.

  • ORION-164301: User quota metrics can now be collected using the /api/prometheusmetrics/all endpoint of VAST Prometheus Exporter.

Platform & Control

  • ORION-164851: Added a validation to ensure that VMS does not allow adding a Mavericks TLC DBox with Rocky Linux-based VAST OS during cluster expansion or node replacement procedures.

  • ORION-167695: Added a mechanism to reboot a cluster's node in case its VAST OS becomes unresponsive, for example, as a result of a boot drive issue. Rebooting the node ensures that the virtual IPs that were assigned to the unresponsive node are no longer kept by the node. This prevents potential duplication of virtual IPs across the cluster, where some of the user requests could be routed to the unresponsive node.

  • ORION-135498: When you choose to disable an SCM drive through VMS, the cluster phases out the drive before deactivating it. Phasing out ensures that at each point in time during the disable process, there always exist two copies of each SCM section, providing for safer cluster operation. During phase-out, the drive state is shown as PHASING_OUT. Once phase-out is complete, the state changes to INACTIVE.

    The phase-out functionality cannot be used for clusters with a single CERES DBox.

Resolved Issues in 5.0.0-SP24

Install & Upgrade

  • ORION-173226: Updated the logic behind EKM port validation to allow specifying port 443 when creating a cluster with external key management through Thales Group CipherTrust Data Security Platform.

  • ORION-163100: Enhanced pre-upgrade validations to prevent failing the upgrade with the failed to ready node before adding to cluster error in case the CNodes have the same product serial.

  • ORION-166035: Enhanced the validation of DNode drives during cluster deployment to ensure that it reports all of the slots where a device is missing or has a different size.

  • ORION-151559: Enhanced upgrade procedures to prevent upgrade failures in situations when cluster nodes are running different VAST Cluster versions (which can be due to a previous failed upgrade, for example).

  • ORION-147505: Improved handling of downloaded release packages so that the user can retry installation without the need to repeat the package download process.

Cluster Expansion

  • ORION-168062: Updated cluster expansion routines to eliminate a failure when trying to add a CERES DBox with the Ceres many files target SCM layout, while the previous layout had many files.

  • ORION-168057: Updated cluster expansion processing to eliminate a flow where previous failed attempts to add a DBox could cause multiple CNode container restarts due to UNKNOWN DNode position.

  • ORION-158006: Provided a field to specify the new IPMI gateway during the cluster expansion procedure so that VMS does not default to the old IPMI gateway, which could cause the DBox add task to fail.

  • ORION-115070: Resolved an issue that prevented completion of the cluster expansion procedure (initiated through VAST Web UI) after adding a new DBox to the cluster.

Networking

  • ORION-174952: Updated VLAN validations to eliminate an issue that could cause a VLAN NOT AVAILABLE IN IB MODE: ObjectModifyResultCode.VLAN_IN_IB_UNSUPPORTED error when trying to add a VLAN tag to a new or existing virtual IP pool on a cluster where all internal ports are InfiniBand.

  • ORION-134785: Updated switch polling routines to eliminate an issue where the cluster raised failed to run mlag_ports, list index out of range alerts while no MLAG ports were configured.

  • ORION-129171: Added a capability to allow using ports other than 636 when configuring LDAP on the VAST cluster.

Element Store

  • ORION-182747: Updated the LDAP caching mechanism to resolve an issue that could cause CNodes to restart with the Buffers pool is exhausted num_blocks=4000 current size=4000 error.

  • ORION-182716: Updated the logic of adaptive chunking to eliminate an issue that could cause an assertion failed: ((batch->get_post_merge_size()) <= (_data2serials.get_max_num_serials_in_batch_before_forced_split())) error followed by an ESTORE MIGRATE deny list alert.

  • ORION-171012: Enhanced defragmentation routines to help prevent flows that could cause the stripe is stuck alerts.

  • ORION-164599: Enhanced cluster behavior upon removal of a directory with a very large number of files to the trash folder so that such a removal, as well as any subsequent removals, can be reflected in quota information without delay. Prior to this change, the quota capacity reclamation tasks could get stuck and the freed capacity could be seen in capacity pages but not in quotas.

  • ORION-156183: Resolved an issue that caused multiple ESTORE TOKEN_SAMPLER deny list alerts with a CNode container restart due to the found a bitmap entry pointing to write buffer in an already migrated snap time error.

  • ORION-156170: Resolved an issue that could, in some cases, cause the target replication cluster not to reclaim space properly upon deletion to the trash folder.

  • ORION-135587: Eliminated an issue that could cause a a TREE_UNLINKER deny list alert due to the assertion failed: (token->_version < (GENY_MAX_VERSION - Globals::geny_manual_recovery_version_buffer - 100)) error.

Quotas

  • ORION-146751: Added a list of blocked users and/or groups to the email notification sent to storage admins when a user quota is exceeded.

NFSv4.1

  • ORION-158487: Resolved an issue where a CNode container restarted due to an error during Nfs4::Nfs4Server::release_connection_state_resources_from_remote_silo(Nfs4::ConnectionState *) processing.

  • ORION-158230: Updated NFSv4.1 lock request processing to eliminate an issue that could cause a CNode container to restart due to the Buffers pool is exhausted error.

SMB

  • ORION-160315: Eliminated a gap in handling of trusted forest’s group SIDs during replication so that it does not cause a STATUS_INSUFF_SERVER_RESOURCES SMB error when using the Enable SMB native authentication option together with async replication.

  • ORION-157632: Resolved an issue that could cause an access denied error when trying to copy a newly created file or directory with a read-only attribute to a VAST SMB share.

  • ORION-146573: Introduced various updates to improve performance when querying for handles inside a directory.

  • ORION-146176: Added a detailed error message to indicate when an attempt to create a directory quota fails because the quota path already has three quotas set along it.

  • ORION-146159: Resolved an issue where upon deletion of a view that had SMB, NFSv3 and NFSv4.1 protocols enabled, the view could still be seen via SMB.

S3

  • ORION-164906: Fine-tuned S3 request processing so that having one very slow connection would not lead to occasional performance drops for other connections where VAST Cluster responds with TCP zero window size notifications.

  • ORION-159628: Optimized processing of CPU-intensive S3 requests to avoid scenarios where they can cause increased cluster latency.

  • ORION-156418: Implemented URL decoding of S3 tags passed as headers in PutObject requests.

  • ORION-150451: Added a capability to configure S3 replication timeouts, helping to fine-tune cluster behavior when interacting with third-party software.

VAST Database

  • ORION-162722: Added logic to properly updated the amount of table rows reported via VMS in case of a transaction rollback.

Replication

  • ORION-176181: Eliminated a flow that could cause a replication failure alert when VAST Cluster attempted to delete a snapshot but its clone was not found because it had already been deleted.

  • ORION-174152: Updated replication to avoid raising the Replication Stream replication missed its RPO target alarm for suspended replication streams.

  • ORION-168683: Made updates to eliminate a flow that could cause a false failed to set destination atime alert to be raised.

  • ORION-167662: Added a meaningful error message in case the replication is stopped due to a missing global snapshot clone.

  • ORION-166445: Introduced a number of enhancements to prevent a scenario where based on existing protection policies, local snapshots were created but were not delivered to the remote site, with many missed PRO alerts reported at the remote site.

  • ORION-156400: Resolved an issue where two internal replication streams were stuck in an INTERNAL_ERROR state.

Authentication & Authorization

  • ORION-162007: Resolved an issue where upon attaching an identity policy to a domain user, VMS did not show the policy when querying the user by username although the policy was attached and worked as expected.

  • ORION-160016: Enhanced the mechanism of merging user group information obtained from multiple providers to ensure that no duplicate group entries are created for a user in the VAST internal database. The duplicate entries could lead to exceeding the user group limit (1024 groups per user), causing access denied in case some of the user groups had to be dropped.

  • ORION-157986: Resolved an issue where an attempt to create an additional S3 key for an Active Directory user which had a historical SID, would fail with a UserDBResultCode.UNEXPECTED_ERROR error.

  • ORION-156632: Updated the Global Catalog (GC) lookup logic to enable VAST Cluster to discover GC servers of the top-level domain if the cluster joined a child Active Directory domain and there were no Global Catalog (GC) servers in the current site.

VMS

  • ORION-165412: Added a caching mechanism to avoid getting a remaining connection slots are reserved for non-replication superuser connections error when the cluster processes a very large amount of metrics requests.

  • ORION-164320: The PEER_IP deny list alerts can now be seen by non-root users, such as the admin user. Prior to this change, these alerts were displayed for root users only.

  • ORION-163880: Resolved an issue that caused the VMS state changed to DEGRADED, reason: CLUSTERED_DB_IS_STOPPING alert when trying to write a file through NFS.

  • ORION-159071: Resolved an issue that caused raising a false mtu is not configured correctly. mtu is 1500 alarm on all CNodes although the MTU was set correctly.

  • ORION-154396: Improved the NVRAM polling mechanism to prevent it from creating extra events that may impact VMS worker performance.

  • ORION-147841: Changed the severity of the switch change state alarm from MAJOR to CRITICAL.

VAST Web UI

  • ORION-161081: Added the Activate and Deactivate options to the actions menu for a protected path (Replication -> Protected Paths -> right-click a path to open the actions menu).

  • ORION-160971: Updated the name of the field used to specify a new column name when renaming a database column in VAST database (DataBase -> VAST DB -> drill down to columns and choose to edit a column) to read Column name instead of Schema name.

  • ORION-160776: When deploying a Sanmina DBox with 30TB disks, GUI messages now include a proper unit of measure for the disk capacity.

  • ORION-159027: When displaying analytics for a view (e.g. having selected a view from the Select Object dropdown in the Analytics page), the Define Time field now shows only options that are applicable to this particular type of analytics.

  • ORION-157902: Updated the logic behind the Name column in the VAST Audit Log page (DataBase -> VAST Audit Log) to always display the log file name.

  • ORION-155650: Updated the logic behind the Add Protected Path dialog to make the Remote Tenant and Remote Path fields non-mandatory for S3 replication.

  • ORION-152204: Provided a more detailed error message in case an invalid value is entered in the Atime Frequence field in view policy advanced settings (Element Store -> View Policies -> choose to create or edit a view policy -> go to Advanced tab).

  • ORION-151892: Removed the Power cycle option from the list of actions available for a slot in the Slots page (Infrastructure -> Slots).

  • ORION-151270: Renamed the following fields in the VAST Easy Install screen to replace the term External with Northbound:

    • Northbound ETH MTU

    • Northbound IB MTU

    • Northbound IB type

  • ORION-147147: Updated the filter for the Link State column on the NICs page (Infrastructure -> NICs) to enable filtering by any of the column’s valid values.

VAST CLI

  • ORION-179969: The --target-id parameter on the replicationstream create command is now optional.

  • ORION-165957: Updated the logic behind the viewpolicy show --audit command to make the command work as expected.

  • ORION-163858: Resolved an issue that caused the --supportbundle --present callhome command to fail with the KeyError: 'upload_kwargs' error.

VAST Prometheus Exporter

  • ORION-169545: Updated VAST Prometheus Exporter to include information about UIDs in user-related metrics (vast_user.*).

Platform & Control

  • ORION-183405, ORION-180739: Introduced updates to eliminate a flow where an NVRAM failure due to an XRQ NVMF backend ctrl timeout error could result in multiple node container restarts, causing temporary service disruption.

  • ORION-182079: Updated the logic of rewriting the data after encryption has been enabled on the cluster to prevent CNodes from restarting with the assertion failed: ((_keys[key_id].get_key_id()) != (NO_OP_KEY_ID)) (0 != 0) key_id=1 encryption_group_id=1 isn't set yet - cannot be used as the current key error in case the cluster had some SSDs in inactive state.

  • ORION-180833: Resolved an issue that could cause multiple CNode containers to restart with the assertion failed: (traversal_mega_shard_id.mega_shard_value() != P::INVALID_SHARD) error.

  • ORION-171091: Resolved an issue that could cause repeated allocated 90% of mooktze buffers! top consumer is TABULAR_TIMEOUT_TICKER alerts for a CNode.

  • ORION-160765: Eliminated a flow that could cause CNode containers to restart with the assertion failed: ((drive.get_size()) > (SSD_BASE_OFFSET)) error in case of a temporary SSD issue.

  • ORION-154985: Eliminated a flow that could cause false BMC firmware mismatch alarms after cluster expansion.

  • ORION-151577: Resolved an issue where multiple CNode containers restarted after deleting a protected path with shard in release for too long and timeout expired for life_type=0,life_gen=<...> (INGEST_READ) errors.

  • ORION-150465: Added more logic to manage timeouts when collecting IPMI sensor logs from Sanmina DNodes to prevent encountering missing data in the logs.

  • ORION-148943: Resolved an issue that could cause a one or more boot-devices are missing error to be reported for a CNode without failing the CNode, with the boot drive being successfully detected within a short time after the alert.

  • ORION-148195: Eliminated an NFS access flow where a race condition could occur, resulting in a CNode container restart with the assertion failed: (!t->vid_uid_link.valid) vid_uid_link should not be valid error.

  • ORION-142529: Resolved an issue where a CNode container restarted with the timeout expired for life_type=1,life_gen=56735750 (INGEST_WRITE) with 1 active jobs error.

  • ORION-137866: Improved handling of traces to prevent situations where some of /vast directories on cluster’s CNodes were used up to 90% if their capacity, with most of data stored being old traces.

Call Home & Support

  • ORION-156360: Resolved an issue that prevented normal SMB log rotation, causing timeouts when attempting to send automatic bundles.

Limitations in 5.0.0-SP24

The following are limitations in VAST Cluster 5.0.0-SP24:

Quotas

  • ORION-208873: Quotas and quota accounting are not supported on subpaths of a replicated protected path on the destination peer. For example, if a protected path is replicated to a destination directory /dest-dir, you cannot set a quota on /dest-dir/mydir.

  • (RESOLVED IN 5.3.0) ORION-179496: NFS aliases are not supported with VAST Cluster's implementation of Remote Quota Protocol (rquota).

Quality of Service

  • ORION-148295: QoS should be enabled on all views to avoid performance degradation issues.

  • ORION-148206: There may be some scenarios in which minimum service levels set by QoS policies are not met. 

  • ORION-139524: Setting a minimum limit for read operations does not limit write operations on the same view.

  • QoS provisioning is not supported for S3 clients.

  • User QoS feature is supported for NFS clients only.

NFS

  • ORION-115336: If one creates an NFSv4.1-only view and mounts it, and then creates its parent view with NFSv3 only, IO operations on the NFSv4.1-only view succeed, but mounts are not allowed.

NFSv3

  • In rare cases with large numbers of files and directories, the existence of a view with Global Synchronization enabled under a protected path can block the removal of the protected path.

SMB

  • ORION-160323: After updating permissions for an SMB share in Windows Explorer, a duplicate SMB share can be displayed. The duplicate SMB share disappears upon a refresh (F5).

  • (RESOLVED IN 5.2.0) ORION-130460: VAST Cluster does not show any previous versions for a file or directory that has the same name as a file or directory that has been deleted and resides in the same directory as the deleted file or directory.

  • ORION-134730: An attempt to restore a file can fail if after the restore has started, a quota is set on the path where the file resides.

  • (RESOLVED IN 5.2.0) ORION-137905: If an application saves changes to a file by recreating the file, or when the client otherwise deletes a file or a directory and creates a new one with the same name, no previous versions can be displayed for the file or directory. To restore such a file or directory, you need to restore one of its parent directories.

S3

  • An object to be uploaded via an S3 presigned POST request must have only ASCII characters in its name.

  • A POST policy (used for S3 presigned POST requests) can be up to 4800 bytes.

VAST Catalog

  • The maximum path length supported by VAST Catalog is 1024 characters.

  • When VAST Catalog is enabled, replication is limited to two peers (group replication is not supported with VAST Catalog). 

  • VAST Catalog must be disabled before a protected path can be deleted. 

Global Snapshot Clones

This release does not support global snapshot clones with VAST Catalog enabled.

Multi-Cluster Management

  • The Multi-Cluster Management feature requires that each cluster participating in the inter-connection is running VAST Cluster 5.0.

  • ORION-135966: The inter-connecting clusters must have connectivity to each other through the clusters' management networks.  

  • ORION-132073: When you remove a VoC cluster from a Multi-Cluster Manager cloud service instance (using the removal button on the cluster's card (delete_voc_button.png)), the VoC cluster is terminated. There is no option to remove a VoC cluster from Multi-Cluster Manager without also terminating it. (In the Multi-Cluster Management page in the VAST Web UI the button removes the VoC cluster from Multi-Cluster Management and does not terminate it. )

  • ORION-137875: In case of Multi-Cluster Manager failure, VoCs provisioned by the instance cannot be connected to a Multi-Cluster Management instance.

Authentication & Authorization

  • ORION-143944: When using Kerberos/NTLM Authentication to authorize SMB users from non-trusting domains, the DOMAIN\username format cannot be used to specify users of remote domains. The username@domain format must be used instead.

  • ORION-134299: When the tenant is set to use Kerberos/NTLM authentication to authorize SMB users from non-trusting domains, both NFS and SMB must use the native SMB authentication (Kerberos), and not Unix-style UID/GIDs.

  • ORION-141763: Before enabling or disabling NTLM authentication, you need to leave the cluster's joined Active Directory domain. After NTLM authentication is enabled or disabled, rejoin the domain.

  • The following limitations apply to Multi-Forest Authentication:

    • VAST Cluster does not allow adding two different Active Directory configuration records with the same domain name but different settings for multi-forest authentication and/or auto-discovery.

    • Names of users' domains are not displayed in data flow analytics.

    • If a trusted domain becomes unavailable and then recovers, SMB clients can use it to connect to the VAST cluster only after a period of time, but not immediately upon domain recovery.

    • Clients cannot establish SMB sessions immediately after a trusted domain recovers from a domain failure.

    • If a group exists on an Active Directory domain in a trusted forest and the group scope is defined as DomainLocal, VAST Cluster does not retrieve such a group when querying Active Directory, so members of such a group are denied access despite any share-level ACLs that can rule otherwise.

    • If TLS is enabled, the SSL certificate has to be a CA-signed certificate that is valid for all of the domain controllers in all trusted forests. If the certificate is not valid for a domain controller, this domain controller is not recognized.

    • ORION-156168: In a multi-forest environment, after migrating a group account from the forest of the cluster’s joined domain to another forest, information about historical group membership is not kept, so users in the migrated group might not be able to access resources to which they used to have access prior to the migration.

VAST Prometheus Exporter

With VAST Cluster 5.0 and 4.7, the Prometheus exporter script at https://github.com/vast-data/vast-exporter is no longer supported. Instead, use the following the VAST API endpoints:

  • https://<VMS IP>/api/prometheusmetrics/ 

  • https://<VMS IP>/api/prometheusmetrics/all 

  • https://<VMS IP>/api/prometheusmetrics/users 

  • https://<VMS IP>/api/prometheusmetrics/defrag 

  • https://<VMS IP>/api/prometheusmetrics/views 

  • https://<VMS IP>/api/prometheusmetrics/devices 

  • https://<VMS IP>/api/prometheusmetrics/quotas 

Call Home & Support

  • When creating a support bundle with the METADATA preset, only one CNode can be selected for the bundle. Selecting any DNode(s) or multiple CNodes together with the METADATA preset results in an error.

Known Issues in 5.0.0-SP24

The following are known issues in VAST Cluster 5.0.0-SP24.

Install & Upgrade

  • ORION-145815: In some cases, VAST Cluster does not raise an alert on a wrong NIC firmware version during a cluster upgrade.

Cluster Expansion

  • ORION-175762: In some cases, a DBox expansion procedure run on a cluster with similarity-based data reduction enabled can take longer than expected.

DBox Replacement

  • (RESOLVED IN 5.0.0-SP30) ORION-167457: If an attempt to run a DBox replacement procedure fails, the Replace option in the DBox actions menu in VAST Web UI (Infrastructure -> DBoxes -> right-click a DBox) remains grayed out, and there is no way to make another attempt. If you encounter this issue, try running the replacement task from VAST CLI using the dbox modify --replace --force command.

Networking

  • ORION-205395: If, during an HA event on a cluster with InfiniBand internal networking, the OpenSM service is found unavailable on a CNode, the CNode may occasionally encounter a failed connecting to the leader's platform error.

  • ORION-155530: Sometimes, after you run the cluster networking configuration script (configure_network.py) and then rebooted the CNode, the eb1 interface can still be down with the Device ib1 has different MAC address than expected, ignoring error. In this case, rerun the script after the reboot to bring the interface up.

Per-Tenant Encryption

  • (RESOLVED IN 5.1.0) ORION-114057: A tenant_create returned an error : ObjectCreateResultCode.FAILURE An error occurs when attempting to create 256 tenants, each with a unique encryption group, if, prior to this attempt, a tenant with per-tenant encryption enabled was created and then deleted.

Quotas

  • (RESOLVED IN 5.2.0) ORION-178975: After creating a user quota with the identifier type set to UID, VMS lists this quota under the corresponding username pulled from the LDAP provider, but not under the UID specified during quota creation.

Lifecycle Rules

  • (RESOLVED IN 5.1.0-SP50, 5.2.0-SP6) ORION-201538: The lifecycle rule mechanism deletes empty directories that were created through NFS and SMB protocols on the view for which a lifecycle rule is enabled, even when the empty directories are not expired according to the enabled lifecycle rule.

QoS

  • ORION-139913: When applying a QoS policy to NFSv3 access, both data and metadata are taken into account in QoS limit calculations, while with NFSv4.1, only data are considered.

  • ORION-137986: Enabling a QoS policy for a view on which a mixed (read and write) workload runs, can result in decreased performance for the workload.

Protocols

  • (RESOLVED IN 5.1.0-SP60, 5.2.0-SP10) ORION-216774: For views with the SMB and S3 protocols enabled and the Mixed Last Wins or SMB security flavor set, the owner of a child directory in a parent that has no default ACL, may in some cases be set incorrectly.

  • (RESOLVED IN 5.3.0) ORION-204972: When creating S3 objects on a multi-protocol view controlled with the NFS security flavor, in a directory for which the SGID POSIX modebit is set, the SGID modebit may get propagated to files/objects created in that directory.

NFS

  • (RESOLVED IN 5.1.0-SP50) ORION-193090: The READDIR and READDIRPLUS operations against a directory with a name longer than 255 characters may hang without returning an error.

  • (RESOLVED IN 5.1.0) ORION-135514: The word percent in the CNode <...> nfs over rdma connections is at <...> percent alert should be read as connections, since the alert shows the number of connections but not a percentage.

SMB

  • (RESOLVED IN 5.3.0) ORION-144020: When the use of Kerberos/NTLM authentication to authorize SMB users from non-trusting domains is enabled for the tenant, a Windows client would let you add a new ACE only by searching for a specific user in the list of trusted forest users, instead of locating the user through the list of domains.

  • ORION-142968: If a quota is exceeded during the process of coping a file to the VAST cluster, the copying process is stopped with a misleading error message: A device attached to the system is not functioning.

S3

  • (RESOLVED IN 5.1.0-SP60) ORION-217661: If the final part of a multipart upload has a size of 0 (zero), VAST Cluster responds with a 400 Bad Request error.

  • (RESOLVED IN 5.3.0) ORION-198606: In rare cases, an IO is stuck - should close alert can be raised on a CNode caused by the cluster waiting for completion of an S3 multi-part upload.

  • (RESOLVED IN 5.1.0) ORION-136816: S3 GET of a symlink is blocked but HeadObject and GetObjectACL operations still succeed.

Protocol Auditing

  • (RESOLVED IN 5.1.0) ORION-134836: When displaying path details in the VAST Audit log dialog, the phandle field does not show the phandle.

VAST Database

  • (RESOLVED IN 5.0.0-SP60) ORION-189280: VAST Cluster does not exclude VAST Database elements from the scope of S3 lifecycle rules. This means that in some cases, database elements can get deleted if the S3 lifecycle rules configured on the cluster stipulate the deletion.

  • ORION-163038: When importing data into a VAST Database table and there is a type mismatch between the column and the data being imported, VAST Cluster produces an ambiguous error message (Failed to get column) instead of pointing to the expected data type.

Data Protection

  • (RESOLVED IN 5.2.0, 5.1.0-SP50) ORION-196575: An attempt to bulk delete a large number of protected paths may result in a timeout in case an issue occurs during deletion of one of the protected paths.

Replication

  • (RESOLVED IN 5.1.0-SP30) ORION-201982: An attempt to replicate from more than eight source clusters may result in a CNode container restart with the Buffers pool is exhausted error.

  • ORION-183432: When trying to perform a failover using the protectedpath modify --modify-replication-state VAST CLI command, the replication state remains Standalone, although it is expected to change from Standalone to Source. If you encounter this issue, use VAST Web UI to perform the failover.

  • (RESOLVED IN 5.2.0) ORION-144137: User quotas for Alternate Data Stream (ADS) children might get miscalculated at the replication destination when the size and/or used attributes of an ADS child are updated due to replication.

  • ORION-140894: When attempting to delete a protected path from the destination peer after an ungraceful failover, a Failed to delete following streams or similar error occurs. The workaround is to manually change the destination peer's role to STANDALONE and retry the deletion.

Multi-Cluster Management

  • (RESOLVED IN 5.1.0) ORION-146029: When sending call home bundles from a VAST on Cloud (VoC) cluster, the Multi-Cluster Manager (MCM) sends the first bundle an hour after the cluster has been registered, and the following bundles are sent according to the user-defined interval.

Authentication & Authorization

  • (RESOLVED IN 5.1.0) ORION-144288: Due to a caching issue, an incorrect user UID can be returned in a user query being retried immediately after the connectivity to the provider has been restored.

VMS

  • (RESOLVED IN 5.0.0-SP60) ORION-206781: A CNode bulk activation task activates only first five CNodes and ignores all the rest nodes in the batch.

  • ORION-203155: The Unexpected width, actual link width is <...>  alarm message may contain garbage at the end of the message.

  • (RESOLVED IN 5.0.0-SP60) ORION-182932: The name of the BW (Mb/s) column in the Global Snapshot Clones page (Data Protection -> Global Snapshot Clones) should read BW (MB/s) to denote megabytes per second.

  • (RESOLVED IN 5.0.0-SP30) ORION-182099: A local variable ‘data’ referenced before assignment error occurs when trying to access predefined analytics reports that provide information per virtual IP pool (for example, VIP Pool Bandwidth).

  • (RESOLVED IN 5.0.0-SP60) ORION-180832: When displayed in VAST Web UI, the definition of the CNode - ProtoMetrics,proto-name=ProtoCommon,latency (ms) event includes ms as a unit of measurement, which is typically used to denote milliseconds. However, in this particular event definition, the threshold is set in microseconds.

  • (RESOLVED IN 5.2.0) ORION-172811: Some analytics properties that can be selected when creating a customized analytics report, produce a graph that does not precisely correspond to the property name. For example, selecting the NFS Write IOPS property produces a graph showing the write IOPS not only for NFS but for all protocols. In particular, this issue may occur with protocol-specific and replication-related properties that represent bandwidth, IOPS and latency.

  • (RESOLVED IN 5.0.0-SP30) ORION-171871: When using VAST Web UI to enable the VMS Preferred option for a virtual IP pool that already exists and includes more than three CNodes, a VMS Preferred must not be True if there are less than 3 CNodes error is shown. To work around this issue, perform this operation using the vippool modify --vms-preferred --cnodes command of VAST CLI.

  • (RESOLVED IN 5.1.0) ORION-147658: An attempt to add a user quota for a non-existing user does not  raise an error.

  • ORION-143717: On a cluster with CNode Port Affinity configured, there is no way to expose the VAST DNS IP on a specific port (left or right).

  • (RESOLVED IN 5.1.0) ORION-134765: The Rows filtered out and Rows scanned metrics in the VAST DB Row Metrics analytics report show the total number of rows accumulated over time while other metrics in the report show the number of rows per second.

  • ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.

  • ORION-89570: In some cases, capacity analytics for subdirectories cannot be reported due to an internal timeout. This issue occurs when there is an extremely large number of subdirectories to be estimated.

VAST Web UI

  • (RESOLVED IN 5.0.0-SP60) ORION-194719: When trying to create a virtual IP pool for all tenants via VAST Web UI (Network Access -> Virtual IP Pools -> choose to create a pool -> select All Tenants in the Tenant field), VAST Cluster creates a pool for the default tenant, instead of creating a pool for all tenants.

  • (RESOLVED IN 5.1.0-SP30) ORION-189217: The Hardware page in VAST Web UI may display a incorrect layout image for a Mavericks DBox.

  • (RESOLVED IN 5.0.0-SP60) ORION-184012: An attempt to export VAST Catalog query results to a CSV file results in a Query: Please correct the form error with no CSV file generated.

  • (RESOLVED IN 5.0.0-SP60) ORION-182932: The name of the BW (Mb/s) column in the Global Snapshot Clones page (Data Protection -> Global Snapshot Clones) should read BW (MB/s) to denote megabytes per second.

  • ORION-169645: A tip for the Atime Frequency field (Element Store -> View Policies -> choose to create or edit a view policy -> General tab) states that 3600s is the default value for this field, while the actual default is 0 (no atime updates).

  • ORION-150503: A local user cannot be found when trying to add it as a value in the Database owner field of the New Database dialog.

  • ORION-147073: The Database page does not show the actual number of rows and size of objects until the page is refreshed manually.

  • (RESOLVED IN 5.1.0) ORION-146832:  After an existing VAST Web UI session has timed out, the Multi-Cluster Management page may display a prompt to enter a registration token for a cluster for which the token has already been provided. To eliminate the prompt, refresh the page.

  • (RESOLVED IN 5.1.0) ORION-146273: After deleting a cluster in the Multi-Cluster Management page, subsequent delete confirmation popups can show the Type DELETE to approve field pre-populated with the DELETE word.

  • (RESOLVED IN 5.1.0) ORION-143724: Some of columns in the SSDs tab of the Infrastructure page opened through Multi-Cluster Management may show dm_mock or mock dev values instead of model and firmware version numbers.

  • ORION-142547: Clicking the Vast catalog policy link in the Policy column of the Snapshots page in Multi-Cluster Management opens an empty Protection Policies page instead of showing a specific policy.

  • (RESOLVED IN 5.1.0) ORION-141670: Relative file symlinks created through SMB are listed as directory symlinks and require use of rmdir to be deleted.

  • (RESOLVED IN 5.2.0) ORION-140652: Auto-completion for the Logon name of the privileged domain user field in tenant settings (Element Store -> Tenants -> choose to create or edit a tenant) is not provided.

  • (RESOLVED IN 5.1.0) ORION-139890: The QoS policy field in the Create View or Update View dialog (Element Store -> Views -> choose to create or edit a view) can list both view QoS policies and user QoS policies, although it does not let you add a user QoS policy to the view.

VAST CLI

  • (RESOLVED IN 5.0.0-SP30) ORION-181077: A cnode add command where a valid value is specified for the --cores parameter may fail with the Illegal arguments: argument --cores: invalid choice error.

  • (RESOLVED IN 5.0.0-SP30) ORION-174959: An attempt to run the protectedpath list --progress command results in a Command Error: Got an unexpected keyword argument 'progress' error.

  • (RESOLVED IN 5.1.0) ORION-146200: The auto-completion options for the role-assign command do not list all possible parameters.

VAST REST API

  • (RESOLVED IN 5.1.0-SP50) ORION-201905: When trying to retrieve the segments retransmitted metrics with an API call to /api/monitors/ad_hoc_query/, a "detail": "metrics not available" error can occur.

  • (RESOLVED IN 5.0.0-SP60) ORION-185217: The /users/<user ID>/ endpoint returns an empty JSON in response to an update request, although the requested updates are made as expected.

  • ORION-178569: The /users/names endpoint always returns only the first 50 entries, regardless of the page size parameter or the total amount of entries to be returned.

Platform & Control

  • ORION-205393: After disconnecting and reconnecting an InfiniBand switch, the cluster might encounter a CNode container restart due to the assertion failed: (!has_verifier(mem_dev->dest().env_id)) Failed performing rpc call! lock_op=HAS_TEMP_REFS error.

  • ORION-203504: A finished redistribution and still not balanced alert can occur on the cluster when one of the CNode ports is disconnected and thus even distribution of virtual IPs among the platform ports is not possible. If there are no accompanying messages indicative of any issues, this alert can be ignored.

  • ORION-202806: When handling extreme workloads, CNode containers may occasionally restart with the timeout expired for life_type=16,life_gen=<number> (TRAVIS) error. The error means that the cluster is busy processing the workload. If there are no other symptoms indicative of any issues, no human intervention is required.

  • (RESOLVED IN 5.3.0) ORION-193956: The leader hogging for <number> us message may occasionally appear in VAST logs. If there are no accompanying messages indicative of a failure, this message can be ignored.

  • (RESOLVED IN 5.1.0-SP60) ORION-158539: The back view for the CERES DBox in the Hardware Layout page shows the data ports in incorrect positions (e.g. port enp3s0f1 is shown on the right while it should be on the left). To mitigate the issue, refer to the Infrastructure -> NICs page that lists the correct locations for the ports.

Call Home & Support

  • ORION-239170: When obfuscating a support bundle, the CNode hostname may not get obfuscated in some of the logs included in the bundle.

  • (RESOLVED IN 5.1.0) ORION-143381: When the directory used to store call home bundles reaches its size cap, a FileNotFoundError: [Errno 2] No such file or directory error is reported instead of an out-of-space error.