VAST Cluster 5.0.0-SP30 Release Notes

Enhancements in 5.0.0-SP30

Resolved Issues in 5.0.0-SP30

Limitations in 5.0.0-SP30

Known Issues in 5.0.0-SP30

ORION-165698: VAST Cluster 5.0.0-SP30 changes the behavior in case no host-based access rules are defined in a view policy:

ORION-172739: Introduced a number of enhancements to VAST traces.

ORION-94711: Added user controls to determine whether to include the root directory (/) in showmount response when the showmount output suppressing functionality is enabled:

ORION-172585: Added the following user controls to view and set the number of splits and subsplits used when processing VAST Database and VAST Catalog queries:

The following legacy user controls have been deprecated:

ORION-174317: The Identity Policies page of VAST Web UI now features a Replicated column that shows whether a policy was replicated from a peer cluster.

ORION-137657: Updated the internal queue size alert so that it is sent to support users only.

ORION-192114: Updated upgrade routines to avoid raising a false Minimal allowed BMC version for DP: "1.52" i s "12.21.7". Target host has BMC ver.: "12.21.1" alarm during the DNode modify task.

ORION-182782: Enhanced upgrade routines to prevent a scenario where, during an upgrade, some of the CNodes cannot be activated due to a missing internal file.

ORION-174960: Made updates to ensure that when running an NDU from VAST CLI, the cnodes_batch_size_percentage option is honored as appropriate.

ORION-178265: For clusters with internal InfiniBand networking, eliminated an issue that could cause OpenSM is not enabled correctly alarms for CNodes that were discovered but were not selected for deployment during a cluster expansion procedure.

ORION-193429: Resolved an issue that could cause an VLAN NOT AVAILABLE IN IB MODE: ObjectCreateResultCode.VLAN_IN_IB_UNSUPPORTED alert on an attempt to create a virtual IP pool with VLAN tagging on a cluster that does not use InfiniBand networking.

ORION-167457: Updated the DBox replacement logic to provide the ability, in VAST Web UI, to retry a DBox replacement procedure if the first attempt has failed.

To retry, navigate to the Dboxes page (Infrastructure -> DBoxes) and choose the Replace option in the actions menu for the DBox you want to replace.

ORION-178078: Eliminated a flow that could CNode containers to restart with the assertion failed: (info>matches == 1) (0 == 1) Bitmap entry doesn't have exactly one matching entry in the content block! counter: 0 error.

ORION-171028: Introduced improvements to eliminate a flow that could cause a significant rise in cluster’s metadata size.

ORION-142232: Resolved an issue that could cause a CNode container to restart with the waiting for a lock for too long error.

ORION-129447: Resolved an issue that could cause reaching halt_writes splittability issue in bitmap alerts followed by high write latency.

ORION-176742: Resolved an issue where after creation of a lifecycle rule, other lifecycle rules were displayed twice in VAST Web UI, and the duplicates could not be removed.

ORION-166711: Improved READDIR request caching to avoid increased latency when handling very large ls requests.

ORION-186969: Added support for creating case-insensitive NFSv4.1 links. Prior to this change, an assertion failed: (case_insensitive == false) case insensitive link?! alert would be raised on such attempts.

ORION-176692: Eliminated a flow that could cause a CNode container to restart with the assertion failed: void Nfs4::ConnectionState::add_session(Nfs4::Session *, Nfs4::Nfs4Metrics *) error.

ORION-172136: Enhanced handling of readdir and getattr requests to eliminate a flow that could cause delays in workload processing.

ORION-164019: Resolved an issue that could cause repeated assertion failed: (handle != INVALID_EHANDLE) can't translate invalid handle: alerts during NFS4 delete-on-close processing.

ORION-159317: Improved processing of bursts of GETATTR and READDIR requests to prevent occasional hanging when a client tries to read a file.

ORION-174206: Updated handling of SMB leases to eliminate a flow that could result in an Existing phandle <...> in lease break map differ from requested phandle alert.

ORION-183657: Updated the logic for handling requests to rename an S3 object so that the object being renamed does not get deleted in case the request fails.

ORION-170605: Updated the output of the VAST CLI's cluster show command to display data and metadata operations separately.

ORION-178354: Made updates to return a correct error code for an attempt to create a database schema on a bucket that already contains S3 objects.

ORION-176490, ORION-167094: Made updates to the logic behind the VAST Prometheus Exporter endpoints to ensure that they provide the same or more capabilities than the deprecated exporter script.

ORION-178082: Updated LDAP query routines to resolve an issue where a custom LDAP domain name could not be resolved for groups, while the resolution worked correctly for individual users.

ORION-172171: Improved the mechanism of querying groups from Active Directory to ensure that all relevant groups are taken into account when running a permission check.

ORION-166221: Added handling of hung OpenLDAP queries so that the cluster is able to automatically recover by restarting the corresponding CNode.

ORION-157997: Improved handling of NIS queries to eliminate an issue that could cause CNode containers to restart with the assertion failed: ((index) < (get_names_max_count())) (50 < 50) Array index out of bounds: QueryName error.

ORION-156145: Resolved an issue that could prevent retrieving the user’s supplementary groups from an LDAP provider.

ORION-153240: Updated the LDAP query caching to eliminate an issue due to which a group query could return different group counts at different times.

ORION-182099: Eliminated an issue that could cause a local variable ‘data’ referenced before assignment error to occur on an attempt to access predefined analytics reports that provide information per virtual IP pool (for example, VIP Pool Bandwidth).

ORION-171871: Resolved an issue that could cause the VMS Preferred must not be True if there are less than 3 CNodes error to appear on attempts to enable the VMS Preferred option for a virtual IP pool that already existed and included more than three CNodes.

ORION-146167: Updated VMS event management to stop writing log messages if the corresponding action has been deactivated in VMS settings.

ORION-159027: When displaying analytics for a view (e.g. having selected a view from the Select Object dropdown in the Analytics page), the Define Time field now shows only options that are applicable to this particular type of analytics.

ORION-135741: Resolved an issue that could cause the VAST Web UI’s Hardware page to show each SSD multiple times with N/A next to it.

ORION-181077: Updated the logic behind the --cores parameter on the cnode add command to prevent the command from failing with the Illegal arguments: argument --cores: invalid choice error when a valid number of cores is specified.

ORION-174959: Updated the protectedpath list command to accept the --progress keyword.

ORION-186081: Resolved an issue where a failure of a DNode with Kioxia drives could lead to CNode container restarts and temporary service disruption.

ORION-176381: Improved management of internal replication-related processes to avoid a flow that could result in CNode containers restarting with the Retry time exceeded - waiting for a free fiber in fiber_group_id=W_RCT_MAINTENANCE for 3600 seconds and timeout expired for life_type=0,life_gen=45431907 (INGEST_READ) with 1 active jobs errors.

ORION-173228: Enhanced the mechanism of automatically disabling NVRAMs on DNode deactivation to eliminate a flow where, after a DNode has failed and then recovered, some of the NVRAMs could remain in NOT_ATTACHED state and had to be activated manually.

ORION-172169: Updated the list of BMC versions supported for Mavericks DNodes used with VAST Cluster 5.0.0 or higher, to avoid raising false DNode BMC Version is not up to date alerts.

ORION-172169: Updated the list of BMC versions supported for Mavericks DNodes used with VAST Cluster 5.0.0 or higher to avoid raising false DNode BMC Version is not up to date alerts.

ORION-157221: Updated port mappings that are used during a cluster installation or expansion procedure with VAST Easy Install to ensure that port position numbers used for clusters with Dell Ice Lake CBoxes correspond to the hardware bezels.

ORION-176953: Updated the logic behind the VAST OS command line alias find-dns-ip to avoid returning too many CNodes.

ORION-208873: Quotas and quota accounting are not supported on subpaths of a replicated protected path on the destination peer. For example, if a protected path is replicated to a destination directory /dest-dir, you cannot set a quota on /dest-dir/mydir.

(RESOLVED IN 5.3.0) ORION-179496: NFS aliases are not supported with VAST Cluster's implementation of Remote Quota Protocol (rquota).

ORION-148295: QoS should be enabled on all views to avoid performance degradation issues.

ORION-148206: There may be some scenarios in which minimum service levels set by QoS policies are not met. 

ORION-139524: Setting a minimum limit for read operations does not limit write operations on the same view.

QoS provisioning is not supported for S3 clients.

User QoS feature is supported for NFS clients only.

ORION-115336: If one creates an NFSv4.1-only view and mounts it, and then creates its parent view with NFSv3 only, IO operations on the NFSv4.1-only view succeed, but mounts are not allowed.

In rare cases with large numbers of files and directories, the existence of a view with Global Synchronization enabled under a protected path can block the removal of the protected path.

ORION-160323: After updating permissions for an SMB share in Windows Explorer, a duplicate SMB share can be displayed. The duplicate SMB share disappears upon a refresh (F5).

(RESOLVED IN 5.2.0) ORION-130460: VAST Cluster does not show any previous versions for a file or directory that has the same name as a file or directory that has been deleted and resides in the same directory as the deleted file or directory.

ORION-134730: An attempt to restore a file can fail if after the restore has started, a quota is set on the path where the file resides.

(RESOLVED IN 5.2.0) ORION-137905: If an application saves changes to a file by recreating the file, or when the client otherwise deletes a file or a directory and creates a new one with the same name, no previous versions can be displayed for the file or directory. To restore such a file or directory, you need to restore one of its parent directories.

An object to be uploaded via an S3 presigned POST request must have only ASCII characters in its name.

A POST policy (used for S3 presigned POST requests) can be up to 4800 bytes.

The maximum path length supported by VAST Catalog is 1024 characters.

When VAST Catalog is enabled, replication is limited to two peers (group replication is not supported with VAST Catalog). 

VAST Catalog must be disabled before a protected path can be deleted. 

The Multi-Cluster Management feature requires that each cluster participating in the inter-connection is running VAST Cluster 5.0.

ORION-135966: The inter-connecting clusters must have connectivity to each other through the clusters' management networks.  

ORION-132073: When you remove a VoC cluster from a Multi-Cluster Manager cloud service instance (using the removal button on the cluster's card ()), the VoC cluster is terminated. There is no option to remove a VoC cluster from Multi-Cluster Manager without also terminating it. (In the Multi-Cluster Management page in the VAST Web UI the button removes the VoC cluster from Multi-Cluster Management and does not terminate it. )

ORION-137875: In case of Multi-Cluster Manager failure, VoCs provisioned by the instance cannot be connected to a Multi-Cluster Management instance.

ORION-143944: When using Kerberos/NTLM Authentication to authorize SMB users from non-trusting domains, the DOMAIN\username format cannot be used to specify users of remote domains. The username@domain format must be used instead.

ORION-134299: When the tenant is set to use Kerberos/NTLM authentication to authorize SMB users from non-trusting domains, both NFS and SMB must use the native SMB authentication (Kerberos), and not Unix-style UID/GIDs.

ORION-141763: Before enabling or disabling NTLM authentication, you need to leave the cluster's joined Active Directory domain. After NTLM authentication is enabled or disabled, rejoin the domain.

The following limitations apply to Multi-Forest Authentication:

https://<VMS IP>/api/prometheusmetrics/ 

https://<VMS IP>/api/prometheusmetrics/all 

https://<VMS IP>/api/prometheusmetrics/users 

https://<VMS IP>/api/prometheusmetrics/defrag 

https://<VMS IP>/api/prometheusmetrics/views 

https://<VMS IP>/api/prometheusmetrics/devices 

https://<VMS IP>/api/prometheusmetrics/quotas 

When creating a support bundle with the METADATA preset, only one CNode can be selected for the bundle. Selecting any DNode(s) or multiple CNodes together with the METADATA preset results in an error.

(RESOLVED IN 5.0.0-SP60) ORION-189672: An attempt to run a VAST Easy Install process with the Northband option selected for the management network fails with the northband is supported only on cnodes and only those with 2 HCAs error. If you encounter this issue, contact VAST Support for a workaround.

ORION-145815: In some cases, VAST Cluster does not raise an alert on a wrong NIC firmware version during a cluster upgrade.

ORION-175762: In some cases, a DBox expansion procedure run on a cluster with similarity-based data reduction enabled can take longer than expected.

ORION-205395: If, during an HA event on a cluster with InfiniBand internal networking the OpenSM service is found unavailable on a CNode, the CNode may occasionally encounter a failed connecting to the leader's platform error.

ORION-155530: Sometimes, after you run the cluster networking configuration script (configure_network.py) and then rebooted the CNode, the eb1 interface can still be down with the Device ib1 has different MAC address than expected, ignoring error. In this case, rerun the script after the reboot to bring the interface up.

(RESOLVED IN 5.1.0) ORION-114057: A tenant_create returned an error : ObjectCreateResultCode.FAILURE error occurs when attempting to create 256 tenants, each with a unique encryption group, if prior to this attempt, a tenant with per-tenant encryption enabled was created and then deleted.

(RESOLVED IN 5.2.0) ORION-178975: After creating a user quota with the identifier type set to UID, VMS lists this quota under the corresponding username pulled from the LDAP provider but not under the UID specified during quota creation.

(RESOLVED IN 5.1.0-SP50, 5.2.0-SP6) ORION-201538: The lifecycle rule mechanism deletes empty directories that were created through NFS and SMB protocols on the view for which a lifecycle rule is enabled, even when the empty directories are not expired according to the enabled lifecycle rule.

ORION-139913: When applying a QoS policy to NFSv3 access, both data and metadata are taken into account in QoS limit calculations, while with NFSv4.1, only data is considered.

ORION-137986: Enabling a QoS policy for a view on which a mixed (read and write) workload runs can result in decreased performance for the workload.

(RESOLVED IN 5.1.0-SP60, 5.2.0-SP10) ORION-216774: For views with the SMB and S3 protocols enabled and the Mixed Last Wins or SMB security flavor set, the owner of a child directory in a parent that has no default ACL, may in some cases be set incorrectly.

(RESOLVED IN 5.3.0) ORION-204972: When creating S3 objects on a multi-protocol view controlled with the NFS security flavor, in a directory for which the SGID POSIX modebit is set, the SGID modebit may get propagated to files/objects created in that directory.

(RESOLVED IN 5.1.0-SP50) ORION-193090: The READDIR and READDIRPLUS operations against a directory with a name longer than 255 characters may hang without returning an error.

(RESOLVED IN 5.1.0) ORION-135514: The word percent in the CNode <...> nfs over rdma connections is at <...> percent alert should be read as connections, since the alert shows the number of connections but not a percentage.

(RESOLVED IN 5.3.0) ORION-144020: When the use of Kerberos/NTLM authentication to authorize SMB users from non-trusting domains is enabled for the tenant, a Windows client would let you add a new ACE only by searching for a specific user in the list of trusted forest users, instead of locating the user through the list of domains.

ORION-142968: If a quota is exceeded during the process of coping a file to the VAST cluster, the copying process is stopped with a misleading error message: A device attached to the system is not functioning.

(RESOLVED IN 5.1.0-SP60) ORION-217661: If the final part of a multipart upload has a size of 0 (zero), VAST Cluster responds with a 400 Bad Request error.

(RESOLVED IN 5.3.0) ORION-198606: In rare cases, an IO is stuck - should close alert can be raised on a CNode caused by the cluster waiting for completion of an S3 multi-part upload.

(RESOLVED IN 5.1.0) ORION-136816: S3 GET of a symlink is blocked, but HeadObject and GetObjectACL operations still succeed.

(RESOLVED IN 5.1.0) ORION-134836: When displaying path details in the VAST Audit log dialog, the phandle field does not show the phandle.

(RESOLVED IN 5.0.0-SP60) ORION-189280: VAST Cluster does not exclude VAST Database elements from the scope of S3 lifecycle rules. This means that in some cases, database elements can get deleted if the S3 lifecycle rules configured on the cluster stipulate the deletion.

ORION-163038: When importing data into a VAST Database table and there is a type mismatch between the column and the data being imported, VAST Cluster produces an ambiguous error message (Failed to get column) instead of pointing to the expected data type.

(RESOLVED IN 5.2.0, 5.1.0-SP50) ORION-196575: An attempt to bulk delete a large number of protected paths may result in a timeout in case an issue occurs during the deletion of one of the protected paths.

(RESOLVED IN 5.1.0-SP30) ORION-201982: An attempt to replicate from more than eight source clusters may result in a CNode container restart with the Buffers pool is exhausted error.

ORION-183432: When trying to perform a failover using the protectedpath modify --modify-replication-state VAST CLI command, the replication state remains Standalone, although it is expected to change from Standalone to Source. If you encounter this issue, use VAST Web UI to perform the failover.

(RESOLVED IN 5.2.0) ORION-144137: User quotas for Alternate Data Stream (ADS) children might get miscalculated at the replication destination when the size and/or used attributes of an ADS child are updated due to replication.

ORION-140894: When attempting to delete a protected path from the destination peer after an ungraceful failover, a Failed to delete following streams or similar error occurs. The workaround is to manually change the destination peer's role to STANDALONE and retry the deletion.

(RESOLVED IN 5.1.0) ORION-146029: When sending call home bundles from a VAST on Cloud (VoC) cluster, the Multi-Cluster Manager (MCM) sends the first bundle an hour after the cluster has been registered, and the following bundles are sent according to the user-defined interval.

(RESOLVED IN 5.1.0) ORION-144288: Due to a caching issue, an incorrect user UID can be returned in a user query being retried immediately after the connectivity to the provider has been restored.

(RESOLVED IN 5.0.0-SP60) ORION-206781: A CNode bulk activation task activates only first five CNodes and ignores all the rest nodes in the batch.

ORION-203155: The Unexpected width, actual link width is <...>  alarm message may contain garbage at the end of the message.

(RESOLVED IN 5.0.0-SP60) ORION-182932: The name of the BW (Mb/s) column in the Global Snapshot Clones page (Data Protection -> Global Snapshot Clones) should read BW (MB/s) to denote megabytes per second.

(RESOLVED IN 5.0.0-SP60) ORION-180832: When displayed in VAST Web UI, the definition of the CNode - ProtoMetrics,proto-name=ProtoCommon,latency (ms) event includes ms as a unit of measurement, which is typically used to denote milliseconds. However, in this particular event definition, the threshold is set in microseconds.

(RESOLVED IN 5.2.0) ORION-172811: Some analytics properties that can be selected when creating a customized analytics report, produce a graph that does not precisely correspond to the property name. For example, selecting the NFS Write IOPS property produces a graph showing the write IOPS not only for NFS but for all protocols. In particular, this issue may occur with protocol-specific and replication-related properties that represent bandwidth, IOPS and latency.

(RESOLVED IN 5.1.0) ORION-147658: An attempt to add a user quota for a non-existing user does not  raise an error.

ORION-143717: On a cluster with CNode Port Affinity configured, there is no way to expose the VAST DNS IP on a specific port (left or right).

(RESOLVED IN 5.1.0) ORION-134765: The Rows filtered out and Rows scanned metrics in the VAST DB Row Metrics analytics report show the total number of rows accumulated over time while other metrics in the report show the number of rows per second.

ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.

ORION-89570: In some cases, capacity analytics for subdirectories cannot be reported due to an internal timeout. This issue occurs when there is an extremely large number of subdirectories to be estimated.

(RESOLVED IN 5.0.0-SP60) ORION-194719: When trying to create a virtual IP pool for all tenants via VAST Web UI (Network Access -> Virtual IP Pools -> choose to create a pool -> select All Tenants in the Tenant field), VAST Cluster creates a pool for the default tenant, instead of creating a pool for all tenants.

(RESOLVED IN 5.0.0-SP60) ORION-184012: An attempt to export VAST Catalog query results to a CSV file results in a Query: Please correct the form error with no CSV file generated.

(RESOLVED IN 5.0.0-SP60) ORION-182932: The name of the BW (Mb/s) column in the Global Snapshot Clones page (Data Protection -> Global Snapshot Clones) should read BW (MB/s) to denote megabytes per second.

ORION-169645: A tip for the Atime Frequency field (Element Store -> View Policies -> choose to create or edit a view policy -> General tab) states that 3600s is the default value for this field, while the actual default is 0 (no atime updates).

ORION-150503: A local user cannot be found when trying to add it as a value in the Database owner field of the New Database dialog.

ORION-147073: The Database page does not show the actual number of rows and size of objects until the page is refreshed manually.

(RESOLVED IN 5.1.0) ORION-146832:  After an existing VAST Web UI session has timed out, the Multi-Cluster Management page may display a prompt to enter a registration token for a cluster for which the token has already been provided. To eliminate the prompt, refresh the page.

(RESOLVED IN 5.1.0) ORION-146273: After deleting a cluster in the Multi-Cluster Management page, subsequent delete confirmation popups can show the Type DELETE to approve field pre-populated with the DELETE word.

(RESOLVED IN 5.1.0) ORION-143724: Some of columns in the SSDs tab of the Infrastructure page opened through Multi-Cluster Management may show dm_mock or mock dev values instead of model and firmware version numbers.

ORION-142547: Clicking the Vast catalog policy link in the Policy column of the Snapshots page in Multi-Cluster Management opens an empty Protection Policies page instead of showing a specific policy.

(RESOLVED IN 5.1.0) ORION-141670: Relative file symlinks created through SMB are listed as directory symlinks and require use of rmdir to be deleted.

(RESOLVED IN 5.2.0) ORION-140652: Auto-completion for the Logon name of the privileged domain user field in tenant settings (Element Store -> Tenants -> choose to create or edit a tenant) is not provided.

(RESOLVED IN 5.1.0) ORION-139890: The QoS policy field in the Create View or Update View dialog (Element Store -> Views -> choose to create or edit a view) can list both view QoS policies and user QoS policies, although it does not let you add a user QoS policy to the view.

(RESOLVED IN 5.1.0) ORION-146200: The auto-completion options for the role-assign command do not list all possible parameters.

(RESOLVED IN 5.1.0-SP50) ORION-201905: When trying to retrieve the segments retransmitted metrics with an API call to /api/monitors/ad_hoc_query/, a "detail": "metrics not available" error can occur.

(RESOLVED IN 5.0.0-SP60) ORION-185217: The /users/<user ID>/ endpoint returns an empty JSON in response to an update request, although the requested updates are made as expected.

ORION-178569: The /users/names endpoint always returns only the first 50 entries, regardless of the page size parameter or the total amount of entries to be returned.

ORION-205393: After disconnecting and reconnecting an InfiniBand switch, the cluster might encounter a CNode container restart due to the assertion failed: (!has_verifier(mem_dev->dest().env_id)) Failed performing rpc call! lock_op=HAS_TEMP_REFS error.

ORION-203504: A finished redistribution and still not balanced alert can occur on the cluster when one of the CNode ports is disconnected and thus even distribution of virtual IPs among the platform ports is not possible. If there are no accompanying messages indicative of any issues, this alert can be ignored.

ORION-202806: When handling extreme workloads, CNode containers may occasionally restart with the timeout expired for life_type=16,life_gen=<number> (TRAVIS) error. The error means that the cluster is busy processing the workload. If there are no other symptoms indicative of any issues, no human intervention is required.

(RESOLVED IN 5.3.0) ORION-193956: The leader hogging for <number> us message may occasionally appear in VAST logs. If there are no accompanying messages indicative of a failure, this message can be ignored.

ORION-239170: When obfuscating a support bundle, the CNode hostname may not get obfuscated in some of the logs included in the bundle.

(RESOLVED IN 5.1.0) ORION-143381: When the directory used to store call home bundles reaches its size cap, a FileNotFoundError: [Errno 2] No such file or directory error is reported instead of an out-of-space error.