Known Issues in 5.4.0-SP3

• (RESOLVED IN 5.4.1-SP5) ORION-329066: An  assertion failed: ((bytes_consumed_by_deserialize) == (serialized_size))  error may occur on an attempt to activate a CNode on a cluster that used to be configured with identity policies in version 5.0 and is now being upgraded to 5.4.

• ORION-299826: When resuming an upgrade of an HPE IceLake CBox with the BMC upgrade flag not set, the upgrade is initiated on all the CNodes, including those that are already at the target version.

• ORION-270710: For some NVIDIA Mellanox NICs, the firmware upgrade process may take much longer than expected. Sometimes, a VMS timeout alert may be raised but the process still completes successfully. If a flint upgrade failure occurs during the process, try power-cycling the node.

• ORION-321148: Duplicate CNode entries may be created in the VMS in case the VMS gets restarted during a CNode add procedure.

• (​​RESOLVED IN 5.4.3​​) ORION-276386: When adding a Ceres v2 DBox to the cluster using the ​dbox add​ command of VAST CLI, the VMS raises a ​HW_VALIDATION - dbox-<...> - None: invalid "dbox min num of hosts". (observed=2, expected: eq 4)​​ alert although the operation completes successfully.

• ORION-220738: In some cases, VMS does not provide any alerts or other status indication when a drive gets disabled while the newly added DBox is being initialized.

• ORION-175762: In some cases, a DBox expansion procedure run on a cluster with similarity-based data reduction enabled can take longer than expected.

DBox Replacement

• ORION-341642: Attempts to run DBox replacement or migration procedures on clusters with both APEX and MLK DNodes fail with the Operation not permitted because following DBoxes are in APEX/MLK mixed state error.

Networking

• ORION-327378: Specifying an incorrect virtual IP configuration when creating or modifying a virtual IP pool may cause a CNode container restart.

• (​​RESOLVED IN 5.4.3​​) ORION-281100: In some cases, a non-zero QoS wait time can be reported for a user while there are no QoS policies in effect for that user. The issue has no impact on the actual IO by the user.

• ORION-236122: Intense read workloads may impact performance on views controlled with a QoS policy with the prioritization flag set.

• ORION-139913: When applying a QoS policy to NFSv3 access, both data and metadata are taken into account in QoS limit calculations, while with NFSv4.1, only data are considered.

• ORION-137986: Enabling a QoS policy for a view on which a mixed (read and write) workload runs, can result in decreased performance for the workload.

NFS

• ORION-318379: When creating subdirectories through NFS, some of the subdirectories may sporadically fail to inherit the default ACL from the parent directory.

• ORION-238708: In some cases, an NFSv4.1 client attempting to move files to the trash folder may get a Permission denied error due to an issue that may cause the trash folder to use a more restricting policy than expected.

• (RESOLVED IN 5.4.1-SP5) ORION-323152: macOS clients may occasionally encounter an ​RPC struct is bad​​ error when moving, deleting or renaming files via SMB, although the operation completes successfully.

• ORION-142968: If a quota is exceeded during the process of coping a file to the VAST cluster, the copying process is stopped with a misleading error message: A device attached to the system is not functioning.

S3

• ORION-345959: The VAST cluster sends message-body data in 304 Not Modified responses, which may cause issues on the client.

• (​​RESOLVED IN 5.4.1-SP5​​) ORION-338771: Processing S3 presigned POST requests with specific contents may occasionally cause a CNode container restart. If you encounter this issue, contact VAST Support for a workaround.

• ORION-329738: Unexpected Content-MD5 header for request type alerts can be raised in case the VAST cluster receives a Content-MD5 header for unexpected request types (e.g. GET). If there are no other symptoms, these alerts can be ignored.

• (​​RESOLVED IN 5.4.1-SP4​​) ORION-329277: Running a VAST cluster with S3 bucket logging enabled may cause existing S3 connections to get stuck and new connections to get rejected due to the cluster's S3 connection limit reached.

• (​​RESOLVED IN 5.4.1-SP5) ORION-324233: The VAST cluster can encounter a flow where CompleteMultipartUpload requests can get stuck, causing the CNodes to reject new S3 connections with an ​IO is stuck - should close connection​​ alert raised.

• (​​RESOLVED IN 5.4.1-SP4​​) ORION-321170: Using S3 SSE-C when uploading objects using chunked encoding may result in unexpected errors.

• (RESOLVED IN 5.4.1-SP4) ORION-284685: In high-load network environments, S3 PutObject requests with larger TLS record sizes (such as those initiated with Boto3 1.40.42 and later) may require several attempts to be processed or encounter connection timeouts.

Block

• (RESOLVED IN 5.4.1-SP4) In some cases, an attempt to perform bulk mapping of block hosts to volumes may return Error 409 Client Error: Conflict for url: <...> Try to create block object that already exist while the VMS is not reporting any existing mappings for the affected volumes.

• ORION-196170: When a parent and child NFSv4.1 view both have same ABAC tags on them, an attempt to mount the child view may result in a Permission denied error. If this occurs, try setting the ABAC tags for the machine account that the client uses to mount the view.

• ORION-336440: When deleting an event topic, VAST Event Broker does not delete the consumer group offsets associated with the deleted topic.

• ORION-330261: In case consumer group members do not perform a graceful disconnect (e.g. do not explicitly leave the group), their member entries are not removed from the VAST cluster, preventing the cluster from providing timely response when other consumers are attempting to join the group.

• (RESOLVED IN 5.4.1) ORION-293900:  When streaming S3 event notifications or VAST DataEngine function-triggering events to a VAST Event Broker view on a cluster with multiple VAST Event Broker views belonging to different tenants, some of the event notifications may encounter a flow that prevents them from being sent to their destination. If this occurs, an Internal Kafka target supported only a single tenant GUID, got 2 alert will be raised. With VAST DataEngine enabled, this issue may result in some functions not being triggered as expected.

• ORION-333926: If the cluster admin generates keys for a local user during the time when the user is logged in to the VAST DataEngine UI (using the old keys), the newly generated keys become valid only after the VMS key cache entry expires. This issue does not occur when the new keys are generated by the users themselves.

• ORION-292066: In VAST DataEngine UI, when trying to connect to a Kubernetes cluster from a VAST tenant that has upper-case letters in its name, the request fails with an Invalid value: <...>: a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character error.

• (RESOLVED IN 5.4.1) ORION-288909: When trying to list logs or traces in VAST DataEngine CLI, the --tenant option on the CLI command does not work as expected.

• ORION-140894: When attempting to delete a protected path from the destination peer after an ungraceful failover, a Failed to delete following streams or a similar error occurs. The workaround is to manually change the destination peer's role to STANDALONE and retry the deletion.

• ORION-145307: Bulk permission updates are not supported for files and directories on satellite clusters.

• ORION-339595: Rotation of Active Directory machine account credentials may in some cases cause subsequent failures in Kerberos authentication of SMB access. If you encounter this issue, unjoin and rejoin the Active Directory domain.

• ORION-335361: Domain discovery failures may occur in a two-forest environment where the trust relationship is established between child domains (one from each forest) but not between the forests' root domains.

• ORION-25479: Latin characters are not supported with LDAP. If you attempt to pass, for example, a username encoded with a Latin character set, the 'LDAP sanity check res: Invalid credentials' error is returned.

• ORION-332874: Cluster's attempts to send email notifications through SMTP may fail with a ​certificate verify failed: self-signed certificate (_ssl.c:1006)​​ error. If you encounter this issue, contact VAST Support for a workaround.

• ORION-329549: In some cases after a cluster upgrade or expansion, the Analytics -> Data Flow and Analytics -> Top Actors pages of VAST Web UI may show the No data to display error instead of actual workload data. If you encounter this issue, contact VAST Support for a workaround.

• ORION-316875: If a virtual IP pool has been renamed, the pool-related metrics obtained through VAST Prometheus Exporter may still show the old pool name.

• ORION-300508: Some metrics may not be available in VAST Prometheus Exporter. Missing metrics include (but may be not limited to) CNode and DNode hardware metrics or CPU temperature, utilization, memory percentage, PCI error count, retransmitted segments, total correctable/uncorrectable memory errors. If you encounter this issue, contact VAST Support for a workaround.

• ORION-293054: VAST Prometheus Exporter exports duplicate entries of ​vast_fan_metrics_hardware_rpm​ and ​vast_fan_active​​ metrics for Ceres v2 DNode fans.

• ORION-292325: When creating a customized analytics report for block volumes or hosts, some metrics intended to reflect a maximum latency are set to a fixed value when there are no block volumes on the cluster.

• (RESOLVED IN 5.4.1) ORION-291686: The read and write latency shown in the Analytics -> Data Flow page of VAST Web UI is calculated without taking into account the amount of I/O on each of the cluster nodes.

• ORION-282603: User-made changes to the ​csi​​ role (one of default administrative roles on the cluster) are not preserved during an upgrade.

• ORION-143717: On a cluster with CNode Port Affinity configured, there is no way to expose the VAST DNS IP on a specific port (left or right).

• ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.

• ORION-89570: In some cases, capacity analytics for subdirectories cannot be reported due to an internal timeout. This issue occurs when there is an extremely large number of subdirectories to be estimated.

• ORION-355178: The ​Acknowledge Filtered​ button in the ​Alarms and Events​​ page does not acknowledge the selected alarms.

• ORION-308654: When using the ​Export selected rows as CSV​​ option of the VAST Web UI, the resulting file may contain all of the rows, including those that were not selected for the export.

• (RESOLVED IN 5.4.1) ORION-290976: When creating an Active Directory or LDAP provider configuration on the VAST cluster, the attribute mapping fields are read-only. To edit these fields, save the provider configuration, then open it for editing and update the fields as needed.

• (​​RESOLVED IN 5.4.1​​) ORION-281960: The ​Activate Write Buffer RAID​ button (​Settings​ -> ​Cluster​ -> ​General Cluster Setup and Actions​ -> ​Write Buffer RAID​​ pane) is not grayed out when the feature has already been activated on the cluster.

• ORION-246284: When entering a value in the Capacity field in the Quota Rules tab in quota settings (Element Store -> Quotas -> choose to create or edit a quota), the UI can unexpectedly alter the value when the user is backspacing or trying to add a decimal point.

• ORION-260502: The Capacity Estimation page (Analytics -> Capacity), which is designed to display capacity for directories or, in case of a VAST Database, for its schema, sometimes also includes VAST Database tables in capacity estimations.

• ORION-239505: The VAST Web UI toggles for selecting operations to be logged in the syslog (Settings -> Notifications -> Syslog Setup) do not enforce logging of the selected operation types as expected. If you encounter this issue, contact VAST Support for a workaround.

• ORION-234835: Some VAST Web UI pages might not allow for proper filtering or sorting by column where value presentation differs from that in the VAST internal database.

• ORION-203189: The External Netmask field in cluster networking settings (Settings -> Configure Network) does not accept alphabetic characters.

• ORION-175189: When querying a local user using the Aggregated context, the Leading GID and Primary group SID fields in the User Details dialog have a value of -1 instead of an empty string.

• ORION-322120: When displaying path names in VAST CLI, the names written in right-to-left languages may appear spelled from left to right.

• ORION-282853: In some cases, changing the view path by using the view modify --path <new path> command does not work as expected. The command may seem to succeed initially but automatically revert back to the original path within minutes.

• ORION-269350: An illegal argument error occurs when trying to run the protectedpath list command with the --protection-policy-name option specified.

• ORION-265720: VAST CLI auto-completion does not include the  --detach-krb-provider option on the tenant modify command.

• ORION-156628: An attempt to run the viewpolicy show --audit command results in a 'ViewPolicyProtocolsAudit' object has no attribute 'get' error.

• ORION-178569: The /users/names endpoint always returns only the first 50 entries, regardless of the page size parameter or the total amount of entries to be returned.

• (​​RESOLVED IN 5.4.3-SP2​​) ORION-355956: In rare cases, the cluster may encounter a high amount of DNode memory ECC errors, causing the leader process to enter repeated failure cycles and resulting in cluster instability.

• (RESOLVED IN 5.4.3-SP1) ORION-353850: A HA event on the cluster's leading node may in some cases result in inability to resume IO after the event due to the new leader not redistributing virtual IPs to the CNodes. This issue can only be encountered with CNodes that are active do not have northbound connectivity (e.g. dual-NIC CNodes where the northdown NIC is down).

• ORION-343501: IO errors occurring on SSDs may cause deactivation of SCMs on the same DNode (and vice versa, errors on SCMs may cause SSD deactivation), resulting in increased IO latency and unnecessary temporary disconnection for the users.

• ORION-344108: Following an upgrade to VAST Cluster 5.4, a DNode HA event may result in multiple container restarts and subsequent service interruption. To avoid encountering this issue, contact VAST Support.

• ORION-327357: For Dell Turin CNodes, the VMS may show the status of the PSUs as ​UNKNOWN​​.

• ORION-255054: Once a DNode replacement operation is complete, the old node may still be listed in the Infrastructure -> DNodes page of VAST Web UI for some minutes.

• ORION-275610: The Drive Compatibility page in VAST Web UI (Support -> Drive Compatibility) does not display all of the details that are available when running a supporteddrives list or supporteddrives get command of VAST CLI.

• ORION-269349: When obfuscating a support bundle, the obfuscation is not applied to files that have already been zipped.

• ORION-239170: When obfuscating a support bundle, the CNode hostname may not get obfuscated in some of the logs included in the bundle.