VAST Cluster 5.0.0-SP70 Release Notes

Enhancements in 5.0.0-SP70

Resolved Issues in 5.0.0-SP70

Limitations in 5.0.0-SP70

Known Issues in 5.0.0-SP70

ORION-192408: Added the ability to control whether the VAST Cluster performs BMC upgrade as part of cluster creation and cluster upgrade procedures. From now on, VMS upgrades BMC and all relevant components (FPGA, BIOS and so on) only if the user has set the BMC upgrade flag.

The following user controls have been added for this purpose:

ORION-204266: Improved handling of BMC firmware versions to eliminate an issue that could cause a BMC matching query does not exist error when trying to add a Mavericks MLK DBox to the cluster.

ORION-203521: Improved NIC link rate validations to avoid generating a false internal nics rate speed is not aligned alarm.

ORION-207892: Resolved an issue that could cause the cluster to encounter a assertion failed: (eres != EStoreRes::OK) ranges_to_mark not empty alert followed by denylisting @handle=<handle> as traversal_type=19 resulted with res=PERSISTENT_ERROR and then locally denylisting shard_type=0 shard_id=<shard> as op=DELETE_ELEMENT resulted with res=28 deny list alerts.

ORION-206865: Enhanced Alternate Data Stream (ADS) handling to eliminate a flow that could result in a CNode container restart with the assertion failed: (!at_least_one_generation_taken) at least one generation type is taken by this fiber or an ancestor fiber in context that expects error.

ORION-204928: Resolved an issue that could cause a false clone_id=1000065870 is MISSING but should have existed alarm associated with a previously deleted replication stream.

ORION-197141: Resolved an issue that could prevent adding new Cumulus 5.9.1 switches in VAST Web UI.

ORION-208873: Quotas and quota accounting are not supported on subpaths of a replicated protected path on the destination peer. For example, if a protected path is replicated to a destination directory /dest-dir, you cannot set a quota on /dest-dir/mydir.

(RESOLVED IN 5.3.0) ORION-179496: NFS aliases are not supported with VAST Cluster's implementation of Remote Quota Protocol (rquota).

ORION-148295: QoS should be enabled on all views to avoid performance degradation issues.

ORION-148206: There may be some scenarios in which minimum service levels set by QoS policies are not met. 

ORION-139524: Setting a minimum limit for read operations does not limit write operations on the same view.

QoS provisioning is not supported for S3 clients.

User QoS feature is supported for NFS clients only.

ORION-115336: If one creates an NFSv4.1-only view and mounts it, and then creates its parent view with NFSv3 only, IO operations on the NFSv4.1-only view succeed, but mounts are not allowed.

In rare cases with large numbers of files and directories, the existence of a view with Global Synchronization enabled under a protected path can block the removal of the protected path.

ORION-160323: After updating permissions for an SMB share in Windows Explorer, a duplicate SMB share can be displayed. The duplicate SMB share disappears upon a refresh (F5).

(RESOLVED IN 5.2.0) ORION-130460: VAST Cluster does not show any previous versions for a file or directory that has the same name as a file or directory that has been deleted and resides in the same directory as the deleted file or directory.

ORION-134730: An attempt to restore a file can fail if after the restore has started, a quota is set on the path where the file resides.

(RESOLVED IN 5.2.0) ORION-137905: If an application saves changes to a file by recreating the file, or when the client otherwise deletes a file or a directory and creates a new one with the same name, no previous versions can be displayed for the file or directory. To restore such a file or directory, you need to restore one of its parent directories.

An object to be uploaded via an S3 presigned POST request must have only ASCII characters in its name.

A POST policy (used for S3 presigned POST requests) can be up to 4800 bytes.

The maximum path length supported by VAST Catalog is 1024 characters.

When VAST Catalog is enabled, replication is limited to two peers (group replication is not supported with VAST Catalog). 

VAST Catalog must be disabled before a protected path can be deleted. 

The Multi-Cluster Management feature requires that each cluster participating in the inter-connection is running VAST Cluster 5.0.

ORION-135966: The inter-connecting clusters must have connectivity to each other through the clusters' management networks.  

ORION-132073: When you remove a VoC cluster from a Multi-Cluster Manager cloud service instance (using the removal button on the cluster's card ()), the VoC cluster is terminated. There is no option to remove a VoC cluster from Multi-Cluster Manager without also terminating it. (In the Multi-Cluster Management page in the VAST Web UI the button removes the VoC cluster from Multi-Cluster Management and does not terminate it. )

ORION-137875: In case of Multi-Cluster Manager failure, VoCs provisioned by the instance cannot be connected to a Multi-Cluster Management instance.

ORION-143944: When using Kerberos/NTLM Authentication to authorize SMB users from non-trusting domains, the DOMAIN\username format cannot be used to specify users of remote domains. The username@domain format must be used instead.

ORION-134299: When the tenant is set to use Kerberos/NTLM authentication to authorize SMB users from non-trusting domains, both NFS and SMB must use the native SMB authentication (Kerberos), and not Unix-style UID/GIDs.

ORION-141763: Before enabling or disabling NTLM authentication, you need to leave the cluster's joined Active Directory domain. After NTLM authentication is enabled or disabled, rejoin the domain.

The following limitations apply to Multi-Forest Authentication:

https://<VMS IP>/api/prometheusmetrics/ 

https://<VMS IP>/api/prometheusmetrics/all 

https://<VMS IP>/api/prometheusmetrics/users 

https://<VMS IP>/api/prometheusmetrics/defrag 

https://<VMS IP>/api/prometheusmetrics/views 

https://<VMS IP>/api/prometheusmetrics/devices 

https://<VMS IP>/api/prometheusmetrics/quotas 

When creating a support bundle with the METADATA preset, only one CNode can be selected for the bundle. Selecting any DNode(s) or multiple CNodes together with the METADATA preset results in an error.

ORION-220709: An error occurs when trying to rerun an OS/FW upgrade on nodes where the firmware has already been staged on the NICs during a prior (unsuccessful) upgrade attempt.

ORION-200435: When running an upgrade with firmware upgrade and force options specified, the firmware does not get upgraded if the DBox is Mavericks APEX.

ORION-145815: In some cases, VAST Cluster does not raise an alert on a wrong NIC firmware version during a cluster upgrade.

ORION-175762: In some cases, a DBox expansion procedure run on a cluster with similarity-based data reduction enabled can take longer than expected.

(RESOLVED IN 5.1.0-SP60) ORION-214087: On a cluster where both external and internal interfaces are InfiniBand, the VMS may report the Failed to find the current OpenSM master with error: 'NoneType' object has no attribute 'ssh_conn' error if the OpenSM master is outside the cluster.

ORION-205395: If, during an HA event on a cluster with InfiniBand internal networking, the OpenSM service is found unavailable on a CNode, the CNode may occasionally encounter a failed connecting to the leader's platform error.

ORION-155530: Sometimes, after you run the cluster networking configuration script (configure_network.py) and then rebooted the CNode, the eb1 interface can still be down with the Device ib1 has different MAC address than expected, ignoring error. In this case, rerun the script after the reboot to bring the interface up.

(RESOLVED IN 5.1.0) ORION-114057: A tenant_create returned an error : ObjectCreateResultCode.FAILURE error occurs when attempting to create 256 tenants, each with a unique encryption group, if prior to this attempt, a tenant with per-tenant encryption enabled was created and then deleted.

(RESOLVED IN 5.1.0-SP60, 5.2.0) ORION-206297: In some cases, quota capacity percentage shown in the Element Store -> Quotas page may not get updated properly to reflect the capacity consumption. If you encounter this issue, use Uplink to view the data.

(RESOLVED IN 5.2.0) ORION-178975: After creating a user quota with the identifier type set to UID, VMS lists this quota under the corresponding username pulled from the LDAP provider but not under the UID specified during quota creation.

(RESOLVED IN 5.1.0-SP50, 5.2.0-SP6) ORION-201538: The lifecycle rule mechanism deletes empty directories that were created through NFS and SMB protocols on the view for which a lifecycle rule is enabled, even when the empty directories are not expired according to the enabled lifecycle rule.

ORION-139913: When applying a QoS policy to NFSv3 access, both data and metadata are taken into account in QoS limit calculations, while with NFSv4.1, only data are considered.

ORION-137986: Enabling a QoS policy for a view on which a mixed (read and write) workload runs, can result in decreased performance for the workload.

(RESOLVED IN 5.1.0-SP60, 5.2.0-SP10) ORION-216774: For views with the SMB and S3 protocols enabled and the Mixed Last Wins or SMB security flavor set, the owner of a child directory in a parent that has no default ACL, may in some cases be set incorrectly.

(RESOLVED IN 5.3.0) ORION-204972: When creating S3 objects on a multi-protocol view controlled with the NFS security flavor, in a directory for which the SGID POSIX modebit is set, the SGID modebit may get propagated to files/objects created in that directory.

(RESOLVED IN 5.1.0-SP50) ORION-193090: The READDIR and READDIRPLUS operations against a directory with a name longer than 255 characters may hang without returning an error.

(RESOLVED IN 5.1.0) ORION-135514: The word percent in the CNode <...> nfs over rdma connections is at <...> percent alert should be read as connections, since the alert shows the number of connections but not a percentage.

(RESOLVED IN 5.3.0) ORION-144020: When use of Kerberos/NTLM authentication to authorize SMB users from non-trusting domains is enabled for the tenant, a Windows client would let you add a new ACE only by searching for a specific user in the list of trusted forest users, instead of locating the user through the list of domains.

ORION-142968: If a quota is exceeded during the process of coping a file to the VAST cluster, the copying process is stopped with a misleading error message: A device attached to the system is not functioning.

(RESOLVED IN 5.1.0-SP60) ORION-217661: If the final part of a multipart upload has a size of 0 (zero), VAST Cluster responds with a 400 Bad Request error.

(RESOLVED IN 5.3.0) ORION-198606: In rare cases, an IO is stuck - should close alert can be raised on a CNode caused by the cluster waiting for completion of an S3 multi-part upload.

(RESOLVED IN 5.1.0) ORION-136816: S3 GET of a symlink is blocked but HeadObject and GetObjectACL operations still succeed.

(RESOLVED IN 5.1.0) ORION-134836: When displaying path details in the VAST Audit log dialog, the phandle field does not show the phandle.

ORION-163038: When importing data into a VAST Database table and there is a type mismatch between the column and the data being imported, VAST Cluster produces an ambiguous error message (Failed to get column) instead of pointing to the expected data type.

(RESOLVED IN 5.2.0, 5.1.0-SP50) ORION-196575: An attempt to bulk delete a large number of protected paths may result in a timeout in case an issue occurs during deletion of one of the protected paths.

(RESOLVED IN 5.1.0-SP30) ORION-201982: An attempt to replicate from more than eight source clusters may result in a CNode container restart with the Buffers pool is exhausted error.

(RESOLVED IN 5.1.0-SP50, 5.2.0) ORION-196091: Objects created as a result of attempts to create a protected path with incorrect settings (for example, to create a path with a target directory that already exists on the destination peer), do not get automatically deleted on protected path creation failure.

ORION-183432: When trying to perform a failover using the protectedpath modify --modify-replication-state VAST CLI command, the replication state remains Standalone, although it is expected to change from Standalone to Source. If you encounter this issue, use VAST Web UI to perform the failover.

(RESOLVED IN 5.2.0) ORION-144137: User quotas for Alternate Data Stream (ADS) children might get miscalculated at the replication destination when the size and/or used attributes of an ADS child are updated due to replication.

ORION-140894: When attempting to delete a protected path from the destination peer after an ungraceful failover, a Failed to delete following streams or similar error occurs. The workaround is to manually change the destination peer's role to STANDALONE and retry the deletion.

(RESOLVED IN 5.1.0) ORION-146029: When sending call home bundles from a VAST on Cloud (VoC) cluster, the Multi-Cluster Manager (MCM) sends the first bundle an hour after the cluster has been registered, and the following bundles are sent according to the user-defined interval.

ORION-196963: The owner for files and folders created on an NFS4.1 view can be occasionally reported as nobody instead of the correct value. This issue can occur if both LDAP and Active Directory providers (with different domain names) are configured for the view’s tenant and the same group exists on both providers, but the user is part of this group on the LDAP provider only.

(RESOLVED IN 5.1.0) ORION-144288: Due to a caching issue, an incorrect user UID can be returned in a user query being retried immediately after the connectivity to the provider has been restored.

(RESOLVED IN 5.2.1) ORION-221021: The CNode write latency event definition contains the word 'read' instead of 'write' in the description text. The event description should read: CNode <CNode> write latency is {threshold} micro seconds.

ORION-203155: The Unexpected width, actual link width is <...>  alarm message may contain garbage at the end of the message.

(RESOLVED IN 5.2.0) ORION-172811: Some analytics properties that can be selected when creating a customized analytics report, produce a graph that does not precisely correspond to the property name. For example, selecting the NFS Write IOPS property produces a graph showing the write IOPS not only for NFS but for all protocols. In particular, this issue may occur with protocol-specific and replication-related properties that represent bandwidth, IOPS and latency.

(RESOLVED IN 5.1.0) ORION-147658: An attempt to add a user quota for a non-existing user does not  raise an error.

ORION-143717: On a cluster with CNode Port Affinity configured, there is no way to expose the VAST DNS IP on a specific port (left or right).

(RESOLVED IN 5.1.0) ORION-134765: The Rows filtered out and Rows scanned metrics in the VAST DB Row Metrics analytics report show the total number of rows accumulated over time while other metrics in the report show the number of rows per second.

ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.

ORION-89570: In some cases, capacity analytics for subdirectories cannot be reported due to an internal timeout. This issue occurs when there is an extremely large number of subdirectories to be estimated.

ORION-209993: The CNode Replication Bandwidth Limit field (Settings -> Cluster -> General) does not display the user-supplied value after saving changes, closing and reopening the dialog, although the value is in effect.

(RESOLVED IN 5.1.0-SP30) ORION-189217: The Hardware page in VAST Web UI may display a incorrect layout image for a Mavericks DBox.

ORION-169645: A tip for the Atime Frequency field (Element Store -> View Policies -> choose to create or edit a view policy -> General tab) states that 3600s is the default value for this field, while the actual default is 0 (no atime updates).

ORION-150503: A local user cannot be found when trying to add it as a value in the Database owner field of the New Database dialog.

ORION-147073: The Database page does not show the actual number of rows and size of objects until the page is refreshed manually.

(RESOLVED IN 5.1.0) ORION-146832:  After an existing VAST Web UI session has timed out, the Multi-Cluster Management page may display a prompt to enter a registration token for a cluster for which the token has already been provided. To eliminate the prompt, refresh the page.

(RESOLVED IN 5.1.0) ORION-146273: After deleting a cluster in the Multi-Cluster Management page, subsequent delete confirmation popups can show the Type DELETE to approve field pre-populated with the DELETE word.

(RESOLVED IN 5.1.0) ORION-143724: Some of columns in the SSDs tab of the Infrastructure page opened through Multi-Cluster Management may show dm_mock or mock dev values instead of model and firmware version numbers.

ORION-142547: Clicking the Vast catalog policy link in the Policy column of the Snapshots page in Multi-Cluster Management opens an empty Protection Policies page instead of showing a specific policy.

(RESOLVED IN 5.1.0) ORION-141670: Relative file symlinks created through SMB are listed as directory symlinks and require use of rmdir to be deleted.

(RESOLVED IN 5.2.0) ORION-140652: Auto-completion for the Logon name of the privileged domain user field in tenant settings (Element Store -> Tenants -> choose to create or edit a tenant) is not provided.

(RESOLVED IN 5.1.0) ORION-139890: The QoS policy field in the Create View or Update View dialog (Element Store -> Views -> choose to create or edit a view) can list both view QoS policies and user QoS policies, although it does not let you add a user QoS policy to the view.

(RESOLVED IN 5.1.0) ORION-146200: The auto-completion options for the role-assign command do not list all possible parameters.

(RESOLVED IN 5.1.0-SP50) ORION-201905: When trying to retrieve the segments retransmitted metrics with an API call to /api/monitors/ad_hoc_query/, a "detail": "metrics not available" error can occur.

ORION-178569: The /users/names endpoint always returns only the first 50 entries, regardless of the page size parameter or the total amount of entries to be returned.

ORION-205393: After disconnecting and reconnecting an InfiniBand switch, the cluster might encounter a CNode container restart due to the assertion failed: (!has_verifier(mem_dev->dest().env_id)) Failed performing rpc call! lock_op=HAS_TEMP_REFS error.

ORION-203504: A finished redistribution and still not balanced alert can occur on the cluster when one of the CNode ports is disconnected and thus even distribution of virtual IPs among the platform ports is not possible. If there are no accompanying messages indicative of any issues, this alert can be ignored.

ORION-202806: When handling extreme workloads, CNode containers may occasionally restart with the timeout expired for life_type=16,life_gen=<number> (TRAVIS) error. The error means that the cluster is busy processing the workload. If there are no other symptoms indicative of any issues, no human intervention is required.

(RESOLVED IN 5.3.0) ORION-193956: The leader hogging for <number> us message may occasionally appear in VAST logs. If there are no accompanying messages indicative of a failure, this message can be ignored.

(RESOLVED IN 5.1.0-SP60) ORION-158539: The back view for the CERES DBox in the Hardware Layout page shows the data ports in incorrect positions (e.g. port enp3s0f1 is shown on the right while it should be on the left). To mitigate the issue, refer to the Infrastructure -> NICs page that lists the correct locations for the ports.

ORION-239170: When obfuscating a support bundle, the CNode hostname may not get obfuscated in some of the logs included in the bundle.

(RESOLVED IN 5.1.0) ORION-143381: When the directory used to store call home bundles reaches its size cap, a FileNotFoundError: [Errno 2] No such file or directory error is reported instead of an out-of-space error.