VAST Cluster 5.4.6 Release Notes

To access this release, contact the VAST Customer Success team via a support ticket, Slack, or by sending an email to support@vastdata.com.

Upgrade to VAST Cluster 5.4.6 is supported from the following releases:

• 5.4.4 - 5.4.4-SP6
• 5.4.3 - 5.4.3-SP5
• 5.4.1 - 5.4.1-SP5
• 5.4.0 - 5.4.0-SP3
• 5.3.5 - 5.3.5-SP5
• 5.3.4 - 5.3.4-SP5
• 5.3.3 - 5.3.3-SP7
• 5.3.2 - 5.3.2-SP8
• 5.3.1 - 5.3.1-SP3
• 5.3.0 - 5.3.0-SP8

Note that direct upgrade may not be supported from hotfix builds. Consult VAST Support regarding upgrade if your cluster is running a hotfix build.

• New Features
• Enhancements
• Resolved Issues
• Limitations
• Known Issues

New Features

• QUALITY OF SERVICE (QOS)

  • S3 Connection Limit per CNode

• PROTOCOL AUDITING

  • Logging Extended Error Information in VAST Audit Log

• AUTHENTICATION AND AUTHORIZATION

  • Provider-Based Authentication of SSH Access to Cluster CNodes

• VMS

  • CA Certificate and VMS IP Binding for SMTP Notifications

• PLATFORM AND CONTROL

  • Monitoring of CNode Liquid Cooling Systems

S3 Connection Limit per CNode

With VAST Cluster 5.4.6, you can limit the number of S3 connections established by a user to any CNode in the cluster by specifying the percentage of CNode connection capacity that the user is allowed to consume. This prevents exhaustion of CNode resources by a single user (or IAM role), which may have impact on other workloads processed by the CNode.

The following user controls let you set the CNode percentage limit:

• In VAST Web UI, the new S3 Client Connections Limit Per CNode (%) pane in user QoS policy settings (Element Store -> QoS Policies -> choose to create or edit a user QoS policy -> User tab)

• In VAST CLI, the --s3-conns-limit-per-cnode-pct option on the qospolicy create and qospolicy modify commands

• In VAST REST API, a new parameter named s3_conns_limit_per_env_pct is accepted in POST requests to the /qospolicies/ endpoint and PATCH requests to the /qospolicies/<ID>/ endpoint.

In previous releases, the number of S3 connections by a user could only be limited for the entire cluster, without taking into account the CNode connection capacity.

Logging Extended Error Information in VAST Audit Log

For S3 requests and and requests to the Security Token Service (STS) that completed with an error, VAST Audit Log can store a detailed explanation of why the request has failed. This helps troubleshoot scenarios where the HTTP error code (which is also available in VAST Audit Log in this release) does not uniquely identify the root cause of the failure.

By default, logging of extended error information is disabled.

The following user controls have been added to enable/disable this feature:

• In VAST Web UI, the Log Extended Error Information option in VAST Audit Log settings (Settings -> Auditing -> General tab -> Global Baseline Audit Settings pane)

• In VAST CLI, the log_extended_error_information keyword for the --audit-options option on the cluster modify, viewpolicy create and viewpolicy modify commands

• In VAST REST API, the new "log_extended_error_information": <true|false> option can be included on the protocols_audit parameter in PATCH requests to the /clusters/<ID>/auditing/ endpoint, POST requests to the /viewpolicies/ endpoint and PATCH requests to the /viewpolicies/<ID>/ endpoint.

Below is an example of a VAST Audit Log entry for a failed PutBucket request with extended error information logging enabled:

AuditEntry(ClusterName='loop1095-kfs', CnodeName='cnode-2', Time='2026-06-28T08:42:42.978Z',
ClientIP='<...>', Host='<...>', ClusterVip='<...>',
VipPoolName='mpt-pool-e341', Protocol='S3', RPCType='PUT_BUCKET', ObjectType='BUCKET',
LoginName='<...>', uid=78290747, S3AccessKeys=['<...>', '', '',
''], Status='TooManyBuckets', HTTPErrorCode='Bad Request',
Path=AuditEntryPath(EHandle='0xffffffffffffffff', CloneID='0xffffffff'), RPCSubTypes=[''],
Tenant='mpt-miserable-ibex', RequestId='0x40110000000b', ConnectionType='HTTP',
BucketName='pjeeueqke-ceigvnewod.xyejwjmnkswdiwn-vzis.fguhqlwnq-dklkmuxgmwo', NumBytes=0,
UsedS3AccessKey='<...>', ErrorDescription='You have attempted to create more buckets than
allowed.')

Provider-Based Authentication of SSH Access to Cluster CNodes

VAST Cluster 5.4.6 supports provider-based authentication when accessing cluster CNodes through SSH.

Users that are members of Active Directory or LDAP group(s) specified for the role will be able to log in to the VAST OS running on CNodes with the same permissions as the vastdata user.

Caution: Consider security implications before granting SSH access to the CNodes.

To take advantage of this feature:

1. Ensure that the cluster has joined Active Directory.

2. Associate an administrative role with one or more Active Directory or LDAP groups for which you want to provide SSH access to the CNodes.

   • In VAST Web UI, open role settings (Administrators -> Administrative Roles -> choose to create or edit a role) and enter the name of the Active Directory or LDAP group in the Active Directory/LDAP groups field. You can supply multiple groups if needed.

3. Configure the role to enable SSH access for to CNodes using Active Directory or LDAP groups:

   • In VAST Web UI, toggle the role's Enable OS SSH Login option on (Administrators -> Administrative Roles -> choose to create or edit a role).

4. Verify that the role create or update task has completed successfully (in the Activities page of VAST Web UI).

The following requirements and limitations apply:

• This feature requires that all CNodes are running VAST OS version 12.15.55-2529260 or later.

• ORION-389075: This feature is not available on DNodes.

CA Certificate and VMS IP Binding for SMTP Notifications

VAST Cluster 5.4.6 lets you upload a CA certificate to be used when sending email notifications on cluster events through SMTP over TLS.

In addition, it provides a new option to send email notifications from the VMS management IP (instead of the IP of the CNode that hosts the VMS).

The following user controls have been added for this purpose:

• In VAST Web UI:

  • The new certificate type SMTP is available in the Settings -> Certificates page for you to upload the certificate to the VMS.

  • The SMTP certificate field next to the Use TLS encryption option in the Settings -> Notifications -> SMTP Setup pane is used to associate an uploaded SMTP certificate with the notification configuration.

  • A new option named Use VMS IP for SMTP under Settings -> Notifications -> SMTP Setup lets you bind the SMTP connection to the VMS management IP.

• In VAST CLI:

  • The --cert-type option of the certificate create and certificate modify commands accepts the new certificate type, SMTP.

  • The eventdefinitionconfig modify command provides new options: --smtp-certificate-id, --smtp-clear-certificate, --smtp-enable-bind-vms-ip, --smtp-disable-bind-vms-ip

• In VAST REST API:

  • POST requests to the /certificates/ endpoint and PATCH requests to the /certificates/<ID>/ endpoint can have their cert_type parameter set to SMTP.

  • PATCH requests to the /eventdefinitionconfigs/<ID>/ endpoint accept the smtp_certificate_id and smtp_bind_vms_ip parameters.

  • GET requests to the /eventdefinitionconfigs/ and /eventdefinitionconfigs/<ID>/ endpoints return the smtp_certificate_name, smtp_certificate_id and smtp_bind_vms_ip parameters.

Monitoring of CNode Liquid Cooling Systems

VAST Cluster 5.4.6 offers new monitoring capabilities for HPE Turin CNodes equipped with liquid cooling systems:

• In VAST Web UI, the new Infrastructure -> Liquid Cooling page provides an inventory of the liquid cooling systems with indication of the box they belong to, overall health status, detailed state and pump information.

• In VAST CLI, the new liquid-cooling list and liquid-cooling show commands.

• In VAST REST API:

  • A GET request to the /liquid-cooling/ endpoint returns a list of liquid cooling systems in the cluster.

  • A GET request to the /liquid-cooling/<ID>/ endpoint returns detailed information about a particular liquid cooling system.

The VMS raises alerts if a liquid cooling system enters a CRITICAL, WARNING or UNKNOWN state, and also when it approaches a maintenance window.

New metrics are available in custom analytics to monitor pump operation, indicating the pump speed, state and days until the next maintenance window. The metrics can be exported with VAST Prometheus Exporter.

Enhancements

Networking

• ORION-348757: Enhanced the cluster networking configuration script (configure_network.py) to provide support for configuring IPMI B2B on a cluster with a northbound external management IP.

Lifecycle Rules

• ORION-287739: Added support for aborting incomplete multipart uploads by enforcing a tag-based lifecycle rule.

S3

• Enhanced S3 checksum verification capabilities as follows:

  • Added support for CRC32, SHA1 and CRC64NVME algorithms.

  • Added support for CopyObject requests with a checksum algorithm that differs from the one used to upload the original object.

  • For example, you can use a CopyObject request with the SHA256 algorithm specified to copy an object that was initially uploaded by a CRC32C-protected PutObject. Prior to this change, such requests would be rejected.

  • The checksum calculated by the cluster is returned in response to the PutObject, CopyObject, UploadPart or POST upload request that had a checksum header in it, and also in response to GetObject and HeadObject requests that contain the x-amz-checksum-mode: ENABLED header.

  Note: S3 checksum verification may cause higher latency per object at the cluster side (due to the increased amount of processing required), and also at clients that are configured to validate checksums returned in response to their GET requests.
  For more information about how the VAST cluster handles S3 checksums, see VAST Cluster Administrator's Guide.

VAST Audit Log

• ORION-360063: Made updates to indicate the RPC status of Success for successful deletions caused by object expiry per lifecycle rules.

• ORION-309404: Added logging of the following information per record:

  • The name of the virtual IP pool through which the request was received

  • For S3 requests, the contents of the HTTP Host header

  • For failed S3 and STS requests, the HTTP error code and, if extended error logging is enabled, the reason for the failure.

Event Publishing

• ORION-386792: VAST Prometheus Exporter can export the following metrics per topic and per view:

  • kafka_external_events_published - number of events published to non-VAST event brokers

  • kafka_external_events_returned_an_error - number of events for which a non-VAST event broker returned an error

  • kafka_external_events_failed_to_send - number of events that could not be sent to non-VAST event brokers.

• ORION-375532: Implemented a keepalive mechanism to monitor connectivity to non-VAST event brokers and raise an alarm in case of connectivity issues.

Authentication and Authorization

• ORION-372295: Starting with VAST Cluster 5.4.6, the VAST STS service accepts requests with the RoleArn parameter in one of the two formats:

  • Recommended format (includes the iam service type): arn:vast:iam::<tenant>:role/<IAM role>

  • Legacy format: arn:vast::<tenant>:role/<IAM role>

  Prior to version 5.4.6, only legacy format was supported.

• ORION-341634: Added support for retrieving user groups' GIDs from the serviceConnectionPoint object class in Active Directory in case the group's gidNumber could not be found in the group object.

• ORION-319553: Added an ability to configure the cluster with an increased number of NIS groups (more than 5120, which is the current limit). To do so, contact VAST Support.

VMS

• Enhanced user controls for managing VAST Web UI idle timeout settings:

  • In VAST Web UI, a new pane named Session Authentication in VMS settings (Settings -> VMS -> General tab) contains the options to enable or disable the automatic logout feature, and also to set the duration of the inactivity period after which the logout occurs.

  • In VAST CLI:

    • The new vms modify_idle_timeout_settings command lets you enable or disable the idle timeout, and also set the inactivity period.

    • The --idle-timeout-settings option on the vms show command shows the current settings for the UI idle timeout.

  • In VAST REST API, the new /vms/<ID>/idle_timeout_settings/ endpoint accepts PATCH and GET requests for configuring the idle timeout and returning the current configuration, respectively.

• ORION-368299: Added predefined analytics reports that show cluster-wide metadata IOPS for NFSv4:

  • Cluster NFSv4 Metadata Read IOPS

  • Cluster NFSv4 Metadata Write IOPS

  • Cluster NFSv4 Control Operations IOPS

• ORION-368089: Added an option to update DataEngine Kafka CA certificate to the right-click menu for a tenant in the Element Store -> Tenants page.

• ORION-362975: The identity policy visual builder (User Management -> Identity Policies -> choose to create or edit an identity policy) now includes a new action category for custom statements under Available Actions -> Database -> Query Engine that lists supported VAST Query Engine actions.

• ORION-362662: Added per-view and per-topic metrics to help monitor and troubleshoot the process of publishing events to non-VAST event brokers:

  • Count of events that were successfully published to the event broker (EventResettledPerKafkaMetrics,..,kafka_external_events_published)

  • Count of events for which the event broker returned an error (EventResettledPerKafkaMetrics,..,kafka_external_events_returned_an_error)

  • Count of events that could not be sent to the event broker (EventResettledPerKafkaMetrics,..,kafka_external_events_failed_to_send)

  The metrics can be obtained for the most recently active 1000 views and 1000 topics.

  Prior to this change, the metrics were available per broker only.

• ORION-327065: Updated the event definition for memory uncorrectable error (UCE) alerts so that the alerts can seen in the VMS by admin users.

VAST Web UI

• ORION-369468: Updated the RAID Status pane to show the RAID rebuild progress as a percentage. (The pane is displayed on top of the cluster's UI when RAID is being rebuilt, for example, as a result of a DBox HA event.)

VAST CLI

• ORION-369783: The --snapshot-id option on the restorepoint list command has been deprecated.

Platform and Control

• ORION-355442: Enhanced nvme_cli log rotation so that the logs are rotated every six hours or at 100MB. This ensures that the logs contain one-week data and consume less than 2GB of space.
• ORION-330539: Introduced enhancements for Ceres v2 DNodes to automatically identify and recover from XFS filesystem errors on the DNode boot drive.

Resolved Issues

Install and Upgrade

• ORION-371034: Resolved an issue where cluster installation could not complete because of the VLAN NOT AVAILABLE IN IB MODE: ObjectCreateResultCode.VLAN_IN_IB_UNSUPPORTED on a none IB cluster error.
• ORION-360772: Resolved an issue where the VMS could report an VAST OS upgrade task as completed while the task was still running.
• ORION-355425: Made updates to ensure that if an upgrade bundle already exists on a node, the upgrade procedure does not re-upload the bundle to that node.
• ORION-339718: Made updates to prevent the upgrade readiness check task from running immediately after the cluster power-up, before the cluster transitions to the ONLINE state.
• ORION-335210: Resolved an issue where an upgrade procedure was performed successfully on the nodes but could not complete the last step due to failed (install_cluster_done): timed out.
• ORION-292431: Eliminated a VAST OS upgrade flow that could result in the upgrade task being stuck in a loop, requiring a manual VMS restart to be completed.
• ORION-284857: Resolved an issue where following an attempt to upgrade a cluster, extra PSU entries were created in the VMS for some of the CNodes with the state of UNKNOWN and incorrect CBox assignment.
• ORION-270092: Made updates to prevent a flow that could cause repeated VMS restarts during an upgrade.
  ORION-254509: Made updates to ensure that the CNodes get empty machine IDs (/etc/machine-id) during installation.

Cluster Expansion

• ORION-370561: Resolved an issue where an attempt to add CBoxes or DBoxes could not succeed due to an error reading 'cboxes' or 'dboxes' in VAST Web UI.
• ORION-327764: Made updates to avoid failing the DBox expansion procedure due to a false Failure domains are imbalanced alert.

Replacements

• ORION-369387: Resolved an issue that could occur after replacement of a CNode in a single-node CBox and cause the VMS to continue reporting the CBox serial number of the previous CNode.
• ORION-293460: Made updates to be able to conclude a DBox replacement procedure if the replace_dbox task times out.

Networking

• ORION-378764: Added support for configuring L3 access (BGP) to ENodes. Prior to this change, L3 could be configured for CNodes only.

Element Store

• ORION-387830: Resolved an issue where an attempt to perform an unsupported VAST Database operation on a replicated bucket caused an ESTORE BIG_CATALOG maintenance denylist alert.
• ORION-386887: Eliminated a race condition that could cause a newman did not find @newman_tid=<...> alert on the cluster.
• ORION-378140: Resolved an issue that could cause the CNode container to restart with the assertion failed: ((__left) == (__right)) (3 == 1) element must have exactly one internal type flag error when performing block handle tracing.
• ORION-375435: Made updates to ensure that the /folders/ endpoints return error code 404 Not Found in case the folder, its path, user or group could not be found when creating a folder, modifying a folder, deleting a folder, or obtaining the folder statistics.
• ORION-369398: Resolved an issue in request queuing logic that could cause high socket latency when processing NFS workloads with QoS enabled.
• ORION-369221: Resolved an issue that could cause high latency when trying to retrieve a list of views from the VMS on a cluster with background tasks from snapshot clone operations.
• ORION-368763: Made updates to improve performance of PATCH requests to the /s3policies/<ID> endpoint.
• ORION-358389: Resolved an issue that could cause the CNode container to restart with the Failed write locking an RWSpinlock for 100000 times - couldn't finalize the write lock error.
• ORION-357291: Enhanced handling of files with a large number of hardlinks to eliminate a flow that could cause very high write latency together with high CNode CPU utilization.
• ORION-357107: Made updates to how the cluster validates bucket policies in case of NFS access to a view controlled with an S3 Native security flavor to avoid a flow that could result in a CNode container restart.
• ORION-354614: Made updates to prevent an SCM hotspot related to similar data blocks in different files that could cause multiple CNode containers to restart with a waiting for a lock for too long error.
• ORION-291701: Resolved an issue that could cause the Too many attributes changes protected by snapshots on handle. IO will fail. Need to delete snapshots to continue alerts on the cluster.

Quotas

• ORION-344126: Made updates to prevent the /quotas/ endpoint from reporting negative values for used effective capacity.

Lifecycle Rules

• ORION-372805: Resolved an issue where the cluster did not delete all of the expired objects per the applicable lifecycle policy.
• ORION-360202: Updated the lifecycle rule logic to handle empty directories created by non-S3 protocols on an S3 bucket view so that they do not get deleted prior to the expiration time as set by the view's lifecycle rule.
• ORION-331067: Enhanced the mechanism of deleting a very large number of objects from an S3 bucket to resolve an issue where objects were not deleted upon expiry based on the configured lifecycle rules.

Quality of Service (QoS)

• ORION-387981: Resolved an issue where a single-view cluster with a very large number of CNodes processing a specific workload, encountered a overflow condition with symptoms appearing as if QoS could not be enforced when the limit was set above 512GB/s.
• ORION-344122: Improved the mechanism of validating S3 workloads against QoS policies to prevent an increase in the cluster's read/write latency that could be encountered due to the QoS check.

Protocols

• ORION-371789: Added an ability to configure the cluster so that NFS clients can list directories that contain absolute path symlinks created via SMB without getting an Invalid argument error for each symlink in the directory.
• ORION-346495: Changed the behavior in case an NFS-enabled view with ACLs disabled (set to BucketOwnerEnforced mode), which is controlled with the S3 Native security flavor, is being accessed by a user which is not the bucket owner. Starting with 5.4.6, the cluster allows traverse access to the bucket in this case. In prior releases, traverse permissions were not granted.

NFS

• ORION-363176: Eliminated a flow where touching a file in a directory that has the SGID bit set, could result in the file's inherited group being overwritten by the user's primary group.
• ORION-357746: Resolved an issue that could, in some cases, cause NFS requests to a snapshot directory to return a permissions error if the snapshot's file or directory being accessed had POSIX ACLs set.
• ORION-354911: Increased the number of host entries that can be returned by the showmount command for a VAST NFS export from 128 to 2048.

SMB

• ORION-378573: Made updates to avoid returning a success code to an SMB client that has requested to delete read-only files on a replication destination path.
• ORION-352674: Improved handling of specific cases where an SMB client closes the connection before receiving a complete response from the VAST cluster, to prevent potential CNode container restarts.
• ORION-351887: Updated the logic that validates the state of the cluster's srvsvc service so that it correctly reports the service state, which allows to automatically restart the service whenever needed.
• ORION-332901: Enhanced processing of SMB QUERY DIRECTORY requests to prevent potential duplication (in the cluster's response) of file_id values across different files, which could cause I/O errors when a macOS client accesses a directory on a VAST SMB share.

S3

• ORION-365253: Eliminated a contention that could occur when processing a specific S3 workload with S3 bucket notifications enabled and cause multiple waiting for a lock for too long, IO is stuck - should close connection and timeout expired for @life_type=16,life_gen=<...> (TRAVIS) errors on the CNodes.
• ORION-360055: Changed the error status code that the VAST cluster returns for S3 requests that fail on connection closure due to a client timeout. Starting with version 5.4.6, the error status code is RequestTimeout. Prior to this change, the cluster returned InternalError.
• ORION-357793: Changed the content of the cluster's 304 NotModified responses to S3 GetObject requests with an IfNoneMatch header so that the response does not include unexpected body data.
• ORION-357559: Improved socket termination logic to avoid unnecessary connection resets when handling encrypted S3 workloads that are continuously slow. In particular, added an ability to configure the internal threshold that determines the size of data that the VAST cluster expects to receive within the stuck socket timeout period so that the connection is not reset due to being stuck. Prior to this change, the threshold was hardcoded to 1MB.
• ORION-353572: Improved validation of object-level POST requests to prevent a flow that could cause the CNode container to restart with the assertion failed: ((val_arg == nullptr) == (val_size_arg == 0)) error.
• ORION-337915: Resolved an issue that could cause ListMultipartUploads requests to intermittently fail due to RequestTimeout with HTTP status code 400 BadRequest.

Block

• ORION-392112: Resolved an issue that could cause CNode containers to restart with signal=11 at address=(nil) code=128 [Non-canonical address (ignore address value, top 16 bits not 0xffff or 0x0000)]) when handling Kafka workloads.
• ORION-385214: Updated the logic used to retry unmapping a block volume to eliminate a condition that could cause INTERNAL DEVICE ERROR due to Reached max simultaneous ctrlrs.
• ORION-385165: Resolved an issue that could cause a read flow without a user ctx guard alert when unmapping block volumes.

Event Publishing

• ORION-390214: Resolved an issue that could cause the CNode container to restart due to the Failed to get kafka params from export error when producing events into a VAST Event Broker topic.
• ORION-380946: Resolved an issue that could occur when creating a VAST event topic with a certain configuration and cause the CNode container to restart.
• ORION-375322: Added a validation to disallow removing the Kafka protocol from a view that contains Kafka topics.
• ORION-375196: Made updates to prevent tenant association-related errors in some scenarios involving deletion and recreation of VAST Event Broker views on multiple tenants.

VAST DataBase

• ORION-354949: Resolved an issue that could cause a CNode container to restart with an assertion failed: ((from) >= (0)) error when processing a request with invalid tabular data.
• ORION-329368: Added logic to avoid indefinite hanging of VastdbApi requests that cannot be processed due to S3 connectivity issues.

VAST DataEngine

• ORION-360940: Added support of output pagination to DataEngine CLI commands.
• ORION-308922: Resolved an issue that could cause a managed application creation task to fail due to a timeout if one of the CNodes selected for the application had the VMS preferred flag set.

Data Protection

• ORION-356416: Updated the bucket policy validation logic to ignore policy statements in cloned or replicated bucket policies where the bucket name is different from the actual name of the destination bucket. This prevents scenarios where the cluster denies access to the destination bucket due to the cloned/replicated bucket policy not automatically updated to use the destination bucket name.
• ORION-345131: Updated the logic used to prefetch data in folders under protected paths to prevent a flow that could cause a You have reached the maximum amount of replication streams error on attempts to create global snapshot clones.
• ORION-342541: Made updates to prevent false alarms on missing PRO targets in case the protection policy has the time to keep local copies set to 0 (zero).

Replication

• ORION-365721: Improved handling of conflicting identity policy names in multi-cluster replication scenarios to always avoid replicating a policy to the cluster where this policy already exists.
• ORION-365285: Eliminated a flow that could cause a replication stream to get stuck in CREATING state with the stream status of INVALID during creation of a remote protected path.

Global Namespace

• ORION-397745: Updated the Global Namespace metadata prefetch logic to prevent a flow that could, in some cases with very large GN paths, result in the SMB clients being unable to open files or folders on a VAST cluster with Global Namespace enabled due to the nv_hash_map is out of space error.
• ORION-378171: Made updates to reduce CPU utilization when deleting a Global Namespace replication stream.

Authentication and Authorization

• ORION-362488: Resolved an issue that could cause false AUTH: Configuration mismatch: uid=<...> mapping changed from sid=<...> to sid=<...> alerts when a local user has the same username and UID as an Active Directory user.
• ORION-351898: Made updates so that the cluster does not attempt to query users from non-VAST authentication providers while the provider discovery is still in progress.
• ORION-319553: Added the ability to configure the cluster with an increased number of NIS groups (more than 5120, which is the current limit). To do so, contact VAST Support.

VMS

• ORION-378582: Added request coalescing logic to improve performance when handling identical concurrent requests to the VMS.

• ORION-378581: Updated the vast_user_connections metric to report the number of user's connections per CNode. Prior to this change, it showed the total number of user's connections per cluster.

• ORION-372335: Resolved an issue where the Data Flow and Top Actors analytics could not be populated with data following an upgrade to VAST Cluster 5.4.

• ORION-372047: Updated the VAST Web UI to prevent the VMS from unexpectedly generating extra alarms based on Threshold or Rate type event definitions, such as informational alarms on snapshot creation.

• ORION-369668: Made updates to prevent duplication of vast_cnode_metrics_SchedulerMetrics entries in VAST Prometheus Exporter output.

• ORION-369407: Resolved an issue that could prevent VAST Prometheus Exporter from exporting values for tenant capacity metrics.

• ORION-366993: Updated the calculations for the User QoS Wait Time metric to ensure the metric provides correct values.

• ORION-366846: Made updates to ensure that quota metrics remain functional if the cluster's VMS has been upgraded to version 5.4 but the CNodes are still running version 5.3.

• ORION-363873: Updated the link speed monitoring logic to eliminate an issue that could, in some cases, cause the VMS to report an incorrect link speed on internal NICs.
  -ORION-363371: Updated the logic behind the Save & Test button in the event definition dialog (Alarms and Events -> Event Definitions -> choose to edit an event definition) so that it substitutes the webhook data variable ${vastdata_cluster_name} with the real cluster name.

• ORION-355322: Resolved an issue that could cause the VMS to get stuck in case it failed over to using Supermicro Gen5 as the box type following a hardware discovery flow failure.

• ORION-350320: Introduced optimizations related to the way the cluster queries its internal database when serving concurrent client requests to reduce the impact on the VMS responsiveness.

• ORION-348701: Updated the logic behind the /prometheusmetrics/user_view/ endpoint to ensure that it works as expected.
  -ORION-346082: Added the metrics to VAST Prometheus Exporter:

  • View metrics:

    • qos_wait_for_budget_time

  • S3 metrics:

    • cmd_wait_parallel_non_write_latency

    • ssec_write_encrypted_data

    • ssec_read_encrypted_data

    • ssec_encryption_bytes

    • ssec_decryption_bytes

• ORION-334818: Changed the logic for sending switch port state alerts so that it sends out alerts for changes from ACTIVE to DOWN. For any other port state transitions (such as from DOWN to INIT, for example), the new logic clears the existing alarm but does not sent out alerts.

• ORION-333283: Made updates to ensure that events with the Alarm only option enabled trigger appropriate email notifications.

• ORION-321410: Made updates so that the VMS reports correct left/right position for all PSUs on Supermicro Turin nodes.

• ORION-300508: Made updates to ensure that VAST Prometheus Exporter exports all expected CNode and DNode hardware metrics, including CPU temperature, utilization, memory percentage, PCI error count, retransmitted segments, total correctable/uncorrectable memory errors.

• ORION-285230: Made updates to prevent a flow that could result in the No data to display error on the Analytics -> Data Flow page of VAST Web UI.

• ORION-256093: Improved validation of VMS TLS certificates being uploaded to the VMS.

VAST Web UI

• ORION-393567: Updated the logic behind the Select Webhook field field in event definition settings (Alarm and Events -> Event Definitions -> choose to edit an event definition -> Action pane) to properly handle removal of a webhook from an event definition. Prior to this change, the removal task appeared to succeed but the webhook was not removed.
• ORION-373776: Made updates to display replicated identity policies in the list of policies that can be/are attached to an IAM role (User Management -> IAM Roles -> choose to create or edit a role -> Attach Identity Policies tab).
• ORION-343049: Made updates to ensure that the VAST Audit Log settings dialog (Settings -> Auditing -> General tab) keeps the selected or deselected state of the Save audit logs to VAST DB and Save audit logs to file checkboxes as set by the user, allowing the user to navigate to other settings pages and return without a change in the checkboxes' state.

VAST CLI

• ORION-369783: Made updates so that the --stream-id and --state options on the restorepoint list command filter the output by the specified stream ID and/or state, respectively.
• ORION-363359: Made updates to the help command of VAST CLI to avoid the KeyError: 'openfile' error when listing the help contents.
• ORION-332126: Updated the lifecyclerule modify command to accept the following options when they are spelled with a hyphen (but not an underscore): --min-size, --max-size, --expiration-days.

Platform and Control

• ORION-387698: Updated handling of failing SCMs by the SCM rebalancing logic to eliminate a flow that could result in multiple CNode container restarts and service interruption.
• ORION-383323: Resolved an issue that could cause increased DNode memory consumption on Ceres v2 clusters.
• ORION-382338: Made updates to prevent CNode containers from restarting with the ((index) < (get_vaids_groups_max_count())) Array index out of bounds: GetUdb2UserResultV2.vaids_groups error when the cluster is running LDAP queries with a large number of group VAIDs involved.
• ORION-379015: Improved the power outage recovery logic to cover a specific scenario that previously required manual intervention to change one of the cluster's internal setting before the automatic recovery procedure could continue.
• ORION-372163: Resolved an issue that could cause the CNode container to restart with the signal=11 at address=<...> ... [Address not mapped to object]) error when handling a SELECT statement for an non-existing VAST Database bucket.
• ORION-370199: Resolved an issue that could cause multiple CNode containers to restart with the Fiber=<...> group=W_GN_BART_SCRUBBER had a duration timeout! error on a cluster that has protected paths with Global Namespace enabled.
• ORION-368612: Resolved an issue that could cause CNode containers to restart on an attempt to calculate Protocol Data Unit (PDU) header digest.
• ORION-365690: Resolved an issue that could cause CNode containers to restart with the assertion failed: ((min_diff) <= (this->_data->get_shard_count())) (97 <= 96) error if the cluster has CNodes that were added to the cluster when the leader was running version 5.2.0 or later.
• ORION-364229: Updated the logic used to maintain information about pre-partitioned drives to prevent a flow where a failure of one drive could cause the box to respond as though it does not contain any active drives.
• ORION-363312: Resolved an issue that could cause the leading CNode container to restart with the memory allocation failed error after DNode activation.
• ORION-358624: Made updates to eliminate a flow where a RAID_SCRUB operation timeout could cause CNode failures and temporary service disruption.
• ORION-354301: Eliminated a race condition that could cause the CNode container to restart with the assertion failed: ((fiber->_join_count) == (0)) (1 == 0) error.
• ORION-343506: Resolved an issue where intensive workload caused some of background maintenance tasks to time out, resulting in multiple CNode container restarts and temporary service disruption.
• ORION-333303: Made updates to prevent false isolcpu is misconfigured alarms on clusters that host CNodes of different types.
• ORION-326533: Resolved an issue that could cause the CNode container to restart with the spinlock lock takes too long error.

Callhome and Support

• ORION-375533: Improved handling of traces when creating a support bundle to prevent collecting memory dumps outside of the time period specified.
• ORION-269349: Added support for obfuscating zipped files in a support bundle.

Limitations

Install and Upgrade

• Automatic drive firmware upgrade is not performed on drives that have been moved manually from an old DBox to a new DBox during the DBox replacement procedure. (Note that this does not apply to EBoxes.)
• (RESOLVED IN 5.5.0) ORION-284784: VAST Cluster Install does not support Subnet field values that end with a dot (for example, 10.10.).
• ORION-281679: SSD firmware upgrade is not supported for Supermicro EBoxes.
• ORION-280966: Having imported a JSON configuration file in VAST Cluster Install, you need to manually verify that all the populated fields have expected values. Sometimes, depending on the cluster configuration and environment, some of the fields are not populated as expected during the import.
• ORION-242658: BMC firmware upgrades are not supported for Supermicro Genoa CNodes.
• ORION-222648: NDU that includes automatic adjustment of CNode CPU isolation settings (isolcpus) is not supported for EBoxes.
• ORION-214559: A BMC upgrade cannot be performed with an inactive CNode that has been powered off.

Networking

• The following limitations apply when using Open Telemetry-Based Ethernet Network Monitoring:
  • Up to 70 switches per cluster.
  • The switch must have OTel configured to send telemetry data over gRPC to the VMS IP address using port 4317.
  • TLS on the connection used to obtain the metrics from the switch is not supported.
  • Supported on NVIDIA Cumulus switches 5.12.1 and later. For other switch vendors, inquire VAST Support.
• The following limitations apply when implementing L3 networking:
  • After enabling L3 access for a virtual IP pool, it cannot be disabled.
  • L3 access is not supported on virtual IP pools for which CNode Port Affinity is configured.
  • ORION-333743: L3 access is not supported on virtual IP pools that include more than one range of virtual IPs.
  • L3 networking is not supported on IB clusters.
  • L3 networking is not available for VAST on Cloud.
  • MD5 authentication is not supported.
  • ORION-266297: When using numbered BGP, each CNode can only be involved in one BGP configuration. To learn which CNodes are currently used in which BGP configurations, use the Network Access -> Virtual IP Pools page that displays CNodes and BGP Configurations for each virtual IP pool.
  • Reverting an L3-configured cluster to L2 through VMS is not supported. If you need to revert, contact VAST Support.
• Changing network configuration from VMS (in VAST Web UI: Settings -> Configure Network) is not supported for clusters that have any CNodes that are connected simultaneously to multiple separate client data networks.
• ORION-374025: Virtual IP pools with a role of Query Engine cannot be modified after creation. This is a built-in safeguard to prevent in-flight queries from failing. If you need to introduce a change into an existing Query Engine pool configuration, delete the pool and create a new one.

Encryption

• The following rules and limitations apply when using Kerberos providers:
  • MIT Kerberos providers are used for NFSv4 only.
  • A tenant can be associated with a single Kerberos provider only.
  • Up to eight Kerberos providers can be defined on a VAST cluster.
  • Kerberos principals used to authenticate to the VAST cluster must exists as users in an LDAP provider or in a VAST provider associated with the same tenant as the Kerberos provider. Otherwise, they are squashed to the nfsnobody user.
• The default tenant does not use keys created or stored within an EKM. To encrypt data within the default tenant with EKM-based keys, use encrypted paths.
• ORION-388667: Creating an encrypted path on a directory exposed by an ABAC-enabled view is not supported. An attempt to create an encrypted path on the same directory as the one exposed by an ABAC-tagged view causes the assigned ABAC tags to be removed.
• ORION-208004: Enabling VAST OS boot drive encryption requires that the node is inactive. Enabling the encryption on an active node may cause a long reboot sequence.

Quotas

• The following limitations apply when using Remote Quota Protocol (rquota):
  • SETQUOTA requests are not supported.
  • Quota information can be provided for active users only.
  • Information about group quotas is not provided.
• Quotas are not enforced on replication destination directories under a protected path. For example, if the protected path is /ppath, a quota on /ppath/yourdir is not enforced.

Lifecycle Rules

• ORION-280461: The maximum size of a lifecycle rule is 15MB.

Quality of Service (QoS)

• Use of QoS with RDMA is not supported.
• The prioritization flag cannot be set for user QoS policies.
• Some high-priority optimizations are applied to NFSv3 only.
• When the cluster-wide maximum write bandwidth is set, the actual performance may be ±15% of the expected performance.
• S3 (including Kafka and VAST Database) and block storage I/Os are not calculated as part of the cluster-wide maximum write bandwidth limit.

NFS

• ORION-324346: NFS aliases are not supported with VAST implementation of Remote Quota Protocol (rquota).
• ORION-115336: If one creates an NFSv4.1-only view and mounts it, and then creates its parent view with NFSv3 only, IO operations on the NFSv4.1-only view succeed but mounts are not allowed.

NFSv4

• TLS encryption with NFSv4.1 is not supported for NFSv.4.1 over RDMA.
• The following requirements and limitations apply when using NFSv4 file delegations:
  • The NFSv4 protocol requires Network Time Protocol (NTP) to be configured on both the VAST cluster and the client. While VAST does not enforce this requirement, the absence of proper time synchronization may, in rare cases, result in unexpected behavior.
  • The client must have an NFSv4 backchannel connection.
  • NFSv4 delegations are not supported with RDMA.
  • For multi-protocol views with NFSv4 delegations enabled, close-to-open consistency is only preserved between NFSv4 clients, but not between NFSv4 clients and other protocols (including NFSv3). For example, if a non-NFSv4 client modifies a file on such a view, the modification may not become immediately visible to an NFSv4 client holding an active delegation on the file.
  • Before making a path read-only by using the /folders/read_only endpoint of VAST REST API, ensure that NFSv4 delegations are disabled or returned. Otherwise, unexpected behavior may be encountered.
  • Restoring a snapshot on a tenant that have NFSv4 delegations enabled, may cause unexpected behavior with regard to delegated files. It is recommended to revoke existing delegations before restoring a snapshot.
  • VAST Cluster recalls granted NFSv4 delegations when a failover process is started.
  • In some cases when there is very high NFSv4 callback load per cluster or per client, VAST Cluster might start recalling or revoking NFSv4 delegations due to callback request timeouts.
  • NFSv4 delegations require mounting the client at least as NFS version 4.1. It is recommended to mount as NFS version 4.2 for additional performance improvements. Linux kernels 6.11+ on a client provide more performance advantages (RFC9754), which get backported to older kernels through VAST-NFS.
  • If you are using VAST-NFS, note that NFSv4 delegations are supported with VAST-NFS 4.0.32 and later. VAST-NFS 4.5 includes enhancements related to support of NFSv4 delegations.
  • SUSE Linux Enterprise Server 12.x and CentOS 7.x clients are not supported with NFSv4 delegations enabled.

SMB

• The following limitations apply when using SMB hardlinks:
  • Creating a hardlink to another SMB-enabled view is not allowed.
  • Creating hardlinks to directories is not allowed.
  • Linking an Alternate Data Stream (ADS) is not supported.
  • Creating a hardlink on a Global Access satellite cluster is not supported.
  • Creating a hardlink, on the Global Access origin cluster, of a file that has already been opened from a satellite cluster, is not allowed.
• The following limitations apply when managing SMB open files:
  • In VAST Global Namespace, operations on open file handles are only supported for the current cluster.
  • ORION-289428: SMB open file queries do not show open SMB hardlink destination files.
  • ORION-288057: When listing files on a path where there is an open ADS stream (path\file:stream), the stream is shown as if the file is a folder (\path\file\stream).
  • For a path with an open Alternate Data Stream (ADS) on it, the following actions are not supported:
    • Closing the ADS
    • Listing individual open handles associated with the ADS.
• Views exposed as SMB shares work only if the cluster is joined to Active Directory. This includes both SMB-only and multiprotocol views.
• The following limitations apply when using SMB change notifications for subdirectories:
  • SMB recursive change notifications are not available for paths for which Global Namespace is configured.
  • Using SMB recursive change notifications for nested SMB shares may entail unexpected behavior.
• The following limitations apply when using Alternate Data Streams (ADS):
  • ADS are not seen or accessible by protocols other than SMB.
  • ADS will not be retained when files and directories are replicated to S3.
    ORION-169707: When the Hyper-V management tool tries to list VAST Hyper-V SMB shares on an SMB server, the The RPC server is not available error can occur if the SMB server is specified using its FQDN. To avoid this error, specify the IP address of the SMB server instead of the FQDN.
• ORION-160323: After updating permissions for an SMB share in Windows Explorer, a duplicate SMB share can be displayed. The duplicate SMB share disappears upon a refresh (F5).
• ORION-134730: An attempt to restore a file can fail if after the restore has started, a quota is set on the path where the file resides.

S3

• The following limitations apply when using S3 Versioning:
  • After versioning has been enabled on a bucket, it cannot be disabled.
  • Multiprotocol access to versioned objects is not supported. NFS and SMB clients may attempt to access the S3 versioned objects in read-only mode.
  • MFA Delete is not supported.
  • S3 conditional writes are not supported for versioned buckets.
  • ORION-288978: S3 versioning is not supported with Global Access.
  • ORION-143808: S3 versioning is not supported with global snapshot clones. An attempt to put a versioned object to a bucket at the global snapshot's destination path fails with an internal error.
• The following rules and limitations apply when using S3 Server-Side Encryption with Customer-Provided Keys (SSE-C):
  • If an S3 request contains an x-amz-server-side-encryption-customer-* header, it is required to use HTTPS for the request to be accepted by the cluster.
  • Only AES256 encryption algorithm is supported.
  • Objects encrypted with SSE-C cannot be accessed through other access protocols.
  • For replication environments, it is strongly recommended to avoid using SSE-C unless all of the clusters involved support SSE-C.
• The following limitations apply when using S3 presigned POST requests:
  • An object to be uploaded via a S3 presigned POST request must have only ASCII characters in its name.
  • A POST policy used for S3 presigned POST requests can be up to 4800 bytes.
• The following limitations apply when using S3 Indestructible Object Mode:
  • An S3 Bucket view with Indestructible Object Mode cannot have other protocols enabled.
  • Indestructible Object Mode cannot be set for a view that points to / (root directory).
  • It is not allowed to have views under the view in Indestructible Object Mode, or at the same path as the Indestructible Object Mode view.
  • Directories that contain views that have indestructible object mode enabled cannot be moved to the trash folder.
  • Indestructible Object Mode cannot be used together with S3 Object Locking or S3 Object Versioning.
  • Indestructible Object Mode cannot be used together with WORM.
  • Indestructible Object Mode cannot be set for a view that exposes the protocol audit log directory.
  • A snapshot cannot be restored on a view that is in Indestructible Object Mode.
  • Views in Indestructible Object Mode are not subject to replication or Global Access.
• ORION-240888: S3 bucket monitoring does not take into account VMS-originated S3 requests, including those related to features such as:
  • Lifecycle rules
  • Event notifications
  • Bucket policies
  • Object ownership
  • Bucket versioning
  • Object locking
  • Bucket logging
• ORION-197281: VAST Cluster disables bucket logging set on a bucket from which data is synchronously replicated to another bucket once you set up bucket logging on the replication destination bucket and configure it to use a different logging destination bucket.
• ORION-190674: Once created, an S3 bucket cannot be renamed or moved to a different path. Thus, for example, if you try to change the bucket’s path when modifying a view in VAST Web UI, the change does not take effect and the view will still be listed with the old path.

Block

• VAST Cluster does not support subsystem names that contain a space (in VAST Web UI: Element Store -> Views -> choose to create a view -> Block tab -> Subsystem name field).
• Snapshots on local protected paths are allowed but replication on non-local protected paths is not supported.
• For Rocky Linux-based clients, VAST recommends that the client uses Rocky Linux 9.4 or later.
• An attempt to remove a volume that has snapshots may cause errors for volume objects of snapshots of that volume, if they exist.
• A view that is used to expose block storage cannot have other storage protocols enabled.
• The following VAST capabilities are not available with block views:
  • Access control features (such as ABE, ABAC, WORM)
  • VAST Audit Log
  • Replication to a remote peer
  • Global Access
  • Remote global snapshot clones
• Nesting of a block view inside an existing block view is not allowed.
• The host NQN cannot be modified. To change the NQN, you need to remove the host and then add and map it anew.
• You cannot enable or disable block storage support on an existing view. Block storage support can only be enabled for a view during view creation and cannot be disabled afterwards.
• Block devices can be created on empty directories only.
• If a host defined on the VAST cluster does not have any volumes mapped to it, NVMe auto-discovery does not show this subsystem.
• When using the VAST Web UI or CLI options to bulk create volumes or hosts, the number of items to be created cannot exceed 256. When mapping hosts to volumes, up to 256 items can be mapped at a time.
• The maximum IO block size is limited to 1MB (4GB for unmap).
• ORION-388015: Compare-and-Write (CAW) operations are only supported when performed against a single block (512B).

Attribute-Based Access Control (ABAC)

• The following features and capabilities cannot be used together with ABAC-tagged views:
  • If a tenant has ABAC-tagged views, you cannot change or remove the Active Directory provider configured for the tenant.
  • When using NFSv4, it is not allowed to create hardlinks in views that have ABAC tags.
  • When using S3:
    • ABAC cannot be used with anonymous S3 access. You cannot set ABAC tags for views that have anonymous S3 access enabled.
    • It is not allowed to set ABAC tags on a view that is a target for S3 bucket logging.
    • Requests from S3 superusers are handled in the same way as for regular users. This means that an S3 superuser is not granted access if the ABAC access check denies access for this user.
  • A directory under which an ABAC-tagged view exists, cannot be moved to the Trash folder.
  • Bulk permission updates are not available for ABAC-tagged views.
  • Lifecycle rules cannot be set for files or directories with ABAC tags.
  • The tenant with ABAC-tagged views cannot have SMB native authentication enabled.
• ABAC is supported on views controlled with SMB, S3 Native and Mixed Last Wins security flavors. ABAC is not supported with NFS flavor.
• ABAC tags cannot be set on the cluster’s root directory (/).
• If a user does not have any ABAC permissions, the user still can mount an NFSv4 export or map a SMB share to a local drive, but the user is not allowed to perform any operations on the files or directories.
• Once assigned, you cannot edit or remove the ABAC tags of a view. Assigning new ABAC tags to an existing view or directory (storage path) is not allowed.
• ABAC is not supported with NFSv3.
• If you create a view for a directory that already exists, ABAC tags from the existing directory are assigned to the newly created view. In this case, there can be a delay between the view creation time and the time when the view's ABAC tags can be displayed.
• After a child view inherits ABAC tags from the parent view, you cannot update or remove the ABAC tags on the child view.
• ORION-388667: Creating an encrypted path on a directory exposed by an ABAC-enabled view is not supported. An attempt to create an encrypted path on the same directory as the one exposed by an ABAC-tagged view causes the assigned ABAC tags to be removed.

Write Once Read Many (WORM)

• You can add additional protocols to a WORM-enabled view, but cannot remove a protocol.
• You cannot configure a WORM-enabled view for NFS/SMB and S3 together.
• Nesting views under a WORM-enabled view is not allowed.
• You cannot disable WORM for the view once it is enabled.
• Bulk permission updates are not allowed on WORM-enabled views.

User Impersonation

• ORION-216379: When VAST protocol auditing is enabled on a user-impersonated view, only UID of the original user is included in the log. The user's login name and SID are not included.

Bulk Permission Updates

• A bulk permission update can run only when the target view (the view exposing the files and directories for which you want to update permissions) is on the same tenant as the template view.
• Only one bulk permission update task per tenant can run at a time.
• Running a bulk permission update on a view where the security flavor does not match that of the template view may result in inaccessible or incompatible permissions set.
• Read-only snapshots and VAST special directories (.vast in S3 buckets, .trash, .snap, .remote) are excluded from bulk permission update.
• If a client attempts to set permissions on directories or files being updated via a bulk permission update, the result is unpredictable.
• Bulk permission update cannot run on ABAC-tagged views.

VAST Audit Log

• ORION-385315: When accessing VAST Audit Log through the VMS REST API (including VAST Web UI), sorting/ordering of VAST Audit Log entries is not supported in case the query results include more than 1000 rows. In this case, VAST Audit Log returns random 1000 entries. To mitigate this behavior, narrow your search to return less than 1000 entries, or use other access mechanisms (such as VAST DB SDK or a query engine connected to the VAST Database).

Event Publishing

• The following limitations apply when using VAST Event Broker:
  • Producer API:
    • Messages are limited to 1MB.
    • In the event record, the key is limited to 126KB and the value is limited to 126KB.
    • Access to topics by UUID is not supported.
    • Idempotent producing is not supported.
    • Automatic creation of topics is not supported.
  • Consumer API:
    • No more than 256 consumer groups per view (broker)
    • The following is not supported:
      • Consumer group stickiness parameters (such as group.instance.id)
      • READ UNCOMMITTED isolation level
      • Cooperative rebalancing
      • Client rack awareness
      • Fetch sessions (only full fetch will be applied), delayed fetch parameters
      • Seek by time
  • Admin API:
    • Supported APIs include the APIs to create topics, delete topics, and to delete groups, as well as describeConfigs and alterConfigs APIs that let you get and update the topic configurations.
  • The following Kafka capabilities are not supported:
    • Over-the-wire compression of messages

    Note: VAST compression of data is supported.

    • Transactions
  • Only one virtual IP pool can be associated with a Kafka-enabled view, providing at least one virtual IP per CNode. Once the view has been created, the virtual IP pool cannot be replaced by another one (but it can be modified if needed).
  • The amount of VAST Event Broker views that you can create on a VAST cluster, is limited by the maximum number of views supported by the cluster and by the maximum number of virtual IP pools (since each broker view requires a dedicated virtual IP pool). See VAST Cluster Scale Guidelines for details.
  • The amount of VAST Event Broker views that you can create on a VAST cluster, is limited by the maximum number of views supported by the cluster.
  • The amount of event topics that you can create on a VAST cluster, is limited by the maximum number of tables per VAST Database table and the overall amount of event topic partitions.
  • A topic can have up to 20,000 partitions. The number of partitions in a topic cannot be changed after the topic has been created. Up to 200,000 partitions are supported per VAST Event Broker view.
  • Event queries based on the topic partition are not supported.
  • When listing consumer groups, the response is limited to 256 groups per Kafka-enabled view.
  • Event publishing and consuming operations, as well as topic management operations are not subject to VAST Protocol Auditing or Quality of Service (QoS).
  • VAST Cluster supports Confluent Kafka Python client 2.4 - 2.8. The aiokafka client is not supported.
  • Authentication and authorization:
    • Active Directory/LDAP is not supported. The user must be defined as a VAST local user.
    • Only one Kafka TLS certificate can be uploaded per VAST cluster.
  • Data protection:
    • The Kafka-enabled view needs to be manually created and associated with a virtual IP pool at the destination peer. The pool must have the same name as the one at the source peer.
    • Fast restore of a protected path containing a Kafka-enabled view is not allowed.
    • VAST replication of consumer groups is not supported. Consumer group offsets are not replicated.

VAST DataBase

• The following limitation applies to vector search:

  • VAST Cluster 5.4 does not include performance optimization for vector operations.

• The VAST connector for Trino does not support the SECURITY clause in CREATE VIEW statements.

• The following limitations apply when using table views:

  • View properties are not supported.
  • Queries to a view must include full table names.
  • Redefining a view is supported for Spark clients only.
  • User-defined column names and comments are lost if the schema of the query changes when redefining a view.

• The following limitations apply to sorted tables:

  • Once enabled for a table, the sorting cannot be disabled.
  • Sorting can be performed for tables that contain 512k or more rows. Although you can enable sorting on a table regardless of its size, smaller tables do not get sorted.
  • Once sorted, tables cannot be unsorted or have their sorted columns changed.
  • Only one sorting order can be defined on a table. Up to four sorted columns are supported.
  • Sorted column values cannot be updated to different values.
  • Sorted tables cannot be replicated. Tables that are replicated cannot be sorted.
  • Sorting cannot be enabled on tables that have semi-sorted projections. Semi-sorted projections cannot be added to sorted tables.
  • Nested data types are not supported for sorted columns.
  • Tables that expose the internal row ID with the vastdb_rowid column cannot be sorted.
  • When calculating the sorting status, a constant value within the Sorted value range is returned for tables that contain less than 5 million rows.

• The following requirements and limitations apply when using row/column-level security:

  • Row filtering and column masking are supported for Trino query engines only.

  • The Trino user's identity policy must have the EndUserImpersonation and GetRowColumnSecurity actions allowed.

• A VAST Database view (a view with the Database protocol) can only be created on clusters with data reduction enabled.

VAST Catalog

• The maximum path length supported by VAST Catalog is 1024 characters.
• ORION-197741: VAST Catalog cannot be enabled on a cluster that uses encryption keys managed through EKM, including per-tenant and per-path encryption keys.

VAST DataEngine

• The following rules and limitations apply when running Trino clusters on VAST:
  • Not less than three CNodes are required per Trino cluster (one for the coordinator, two or more for the workers).
  • In case of CNode HA events, the Trino cluster remains accessible and running queries as long as one coordinator CNode and one of the worker CNodes are up.
  • A VAST cluster upgrade (NDU) performed while running a Trino cluster may cause Trino query failures. The Trino cluster will be fully operational after the NDU is complete.
• If VAST DataEngine is enabled on the cluster, replication of the tenant's root directory is not allowed.
• Up to 32 engines are supported per VAST cluster.

Data Protection

• When using NFSv3, in rare cases with large numbers of files and directories, the existence of a view with Global Synchronization enabled under a protected path can block the removal of the protected path.

Replication

• The following limitation applies to VAST Database asynchronous replication:
  • ORION-179909: VAST Database asynchronous replication cannot be used together with Global Access or synchronous replication on the same path.
  • Only committed database transactions are replicated.
  • VAST Catalog and audit log are not replicated.
  • Multi-tenant replication of VAST Databases is not supported.
• The following limitations apply to synchronous replication:
  • A protected path using synchronous replication can only contain S3 buckets. You cannot use synchronous replication for paths that are exposed to any other protocols.
• The following limitations apply to synchronous replication for S3:
  • Synchronous replication is supported for S3 buckets only.
  • It is not allowed to configure local snapshots, global snapshot clones or Global Access on the protected path for which synchronous replication is configured.
  • S3 lifecycle rules are not replicated.
  • S3 keys are replicated asynchronously.
  • Synchronously replicated directories are not subject to bulk permission updates.
• Protected paths with asynchronous replication cannot be nested.
• Data cannot be moved into or out of a path that is protected by either asynchronous replication or S3 replication.
• it is not possible to change which protection policy controls any replication stream.
• ORION-208123: Local user accounts are not subject to replication.

Global Access

• VAST Database and VAST Event Broker (Kafka) views are not supported.
• The following limitations apply when using Global Access for S3 buckets:
  • Identity policies must be enabled at the cluster to which they get replicated.
  • The following VAST capabilities are not supported on destination buckets:
    • S3 event notifications
    • S3 Indestructible Object Mode
    • Lifecycle policies
    • Write Once Ready Many (WORM)
  • Bucket logging is only supported if both the source and destination buckets are in the same protected path.
  • Bucket replication between two clusters is only supported when the bucket is associated with the default S3 view policy.
  • S3 endpoints are not replicated.
• The following combinations of access protocols and security flavors are supported:
  • Access to a Global Access protected path (global folder) on the source cluster: NFSv3, SMB, S3 with any of the flavors; NFSv4 with any flavor except the SMB flavor.
  • Access to a satellite path on the destination cluster: NFSv3 with NFS or S3 Native flavor, SMB (all flavors), S3 (all flavors)
  • Access to a path which is part of both Global Access and asynchronous replication setup: NFSv3 with NFS flavor, SMB (all flavors), S3 (all flavors)
    If a view is configured with both NFSv4 and SMB, it must be controlled with the NFS security flavor.
• Lease expiration time can only be set when creating a global access protected path. You cannot change lease expiration time when you modify a global access path.
• VAST Catalog does not provide information on the cached data on the remote cluster.
• ORION-194805: Applications that use SMB2 Byte Range Locks are not supported when the SMB client is connected via a remote Global Access protected path. Examples of such applications are Microsoft Office suite on macOS, Microsoft Hyper-V, AutoDesk 3ds Max and some Adobe Premiere plugins.
• ORION-194613: If some files have additional hardlinks, the amount of bytes reported as prefetched can be higher than the actual amount prefetched.

VAST DataSpace

• VAST DataSpace requires that each cluster participating in the inter-connection is running VAST Cluster 5.0 or later.
• VAST clusters with IPv6 management IPs cannot be added to VAST DataSpace. Only IPv4 is supported.
• ORION-135966: The inter-connecting clusters must have connectivity to each other through the clusters' management networks.

VAST on Cloud (VoC)

• VAST Cloud (Polaris) UI does not provide an option to run a cluster upgrade. Cluster upgrades are to be performed using the VAST Web UI.
• VAST on Cloud clusters do not support OS upgrade.
• In the event of downtime, data is rebuilt while the cluster comes back online. In case of a subsequent failure during the rebuild, data integrity is not guaranteed.
• ORION-327048: For VoC on GCP, it is recommended to have a multiple of 8 non-VMS VMs per cluster, since GCP has 8 failure domains and imbalance may result in failed stripe allocations.
• ORION-145141: Creating a tenant with EKM encryption is not supported on VoC clusters.
• ORION-113036: After you reregister the same VoC cluster in Uplink, information about the previously registered instance of this cluster is no longer available in Uplink.

Authentication and Authorization

• The following rules and limitations apply when using STS, IAM roles and OIDC providers:

  • STS is supported with HTTPS only.
  • Use of STS requires that the S3 protocol is enabled for the cluster.
  • One OIDC provider per VAST cluster tenant.
  • When replicating IAM roles:
    • ORION-278747: IAM role replication may take several minutes to complete in case the cluster is under high load or there is a very large amount of roles to replicate.
    • STS access keys are replicated only when the protection path is in SYNC state.
    • ORION-278217: When processing the assume role operation, the cluster may be raising a Failed to replicate STS temp keys alert until the synchronous replication stream/protection path reaches the SYNC state.
  • ORION-287563: Only five OIDC keys can be returned for a manual request to get or refresh OIDC keys.

• The following requirements and limitations apply when enabling OS SSH login:

  • This feature requires that all CNodes are running VAST OS version 12.15.55-2529260 or later.
  • ORION-389075: This feature is not available on DNodes.

• The following limitations apply when using netgroups:

  • Hosts should have both forward and reverse DNS entries. When VAST Cluster gets the netgroup hostname response from a NIS or LDAP server, it resolves the hostname via DNS.
  • Netgroups are only used to allow or deny clients' access via NFS. VAST Cluster does not accept netgroup entries in host-based access rules for other access protocols.

• The following limitations apply to Multi-Forest Authentication:

  • VAST Cluster does not allow adding two different Active Directory configuration records with the same domain name but different settings for multi-forest authentication and/or auto-discovery.
  • Names of users' domains are not displayed in data flow analytics.
  • If a trusted domain becomes unavailable and then recovers, SMB clients can use it to connect to the VAST cluster only after a period of time, but not immediately upon domain recovery.
  • Clients cannot establish SMB sessions immediately after a trusted domain recovers from a domain failure.
  • If a group exists on an Active Directory domain in a trusted forest and the group scope is defined as DomainLocal, VAST Cluster does not retrieve such a group when querying Active Directory, so members of such a group are denied access despite any share-level ACLs that can rule otherwise.
  • If TLS is enabled, the SSL certificate has to be a CA-signed certificate that is valid for all of the domain controllers in all trusted forests. If the certificate is not valid for a domain controller, this domain controller is not recognized.
  • ORION-156168: In a multi-forest environment, after migrating a group account from the forest of the cluster’s joined domain to another forest, information about historical group membership is not kept, so users in the migrated group might not be able to access resources to which they used to have access prior to the migration.

• The following limitations apply when using Kerberos/NTLM authentication:

  • ORION-143944: When using Kerberos/NTLM Authentication to authorize SMB users from non-trusting domains, the DOMAIN\username format cannot be used to specify users of remote domains. The username@domain format must be used instead.
  • ORION-141763: Before enabling or disabling NTLM authentication, you need to leave the cluster's joined Active Directory domain. After NTLM authentication is enabled or disabled, rejoin the domain.
  • ORION-134299: When the tenant is set to use Kerberos/NTLM authentication to authorize SMB users from non-trusting domains, both NFS and SMB must use the native SMB authentication (Kerberos), and not Unix-style UID/GIDs.

• The following limitations apply when using an allow-list of Active Directory DCs and GCs for LDAP queries:

  • The list must include at least one Global Catalog.
  • The list can contain up to 10 entries.
  • The Active Directory provider must have multi-forest authentication disabled.

• ORION-202335: If the cluster has Active Directory domain auto-discovery enabled, the discovered domains are kept in cache for quite a long time. If you modify an existing provider's configuration while auto-discovery is on, VMS may still report the old cached entries. To avoid this, rerun auto-discovery or remove and re-add the provider.

• ORION-195524: Following a cluster recovery and while the Active Directory provider is still inaccessible, VAST Cluster can resume IO of provider users if they use NFSv3 or NFSv.4.1 with NTLM authentication. IO of provider users accessing through SMB or NFSv4.1 with Kerberos authentication is not resumed during this period.

• ORION-187136: Identity policies are replicated as disabled to the destination peer, where if needed, they can be enabled manually.

• ORION-152475: An access denied error is returned for NFSv3 or NFSv4 requests if they are checked against an identity or bucket policy with an s3:ExistingObjectTag condition statement in it.

VMS

• Tenant client metrics can only be collected for NFSv3 and NFSv4.
• The following limitations apply when installing a SSL certificate for VMS:
  • Only RSA-generated public keys are supported.
  • Password-protected private keys are not supported.
• When setting a VMS login banner text via the VAST CLI, multiple lines are not supported.
• ORION-212118: If a wrong VMS token is passed, the cluster responds with 403 FORBIDDEN but not with 401 UNAUTHORIZED.
• ORION-187584: An empty realm (which does not contain any objects) cannot be assigned to a role.
• ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.

VAST Web UI

• The visual identity policy builder in VAST Web UI (User Management -> Identity Policies -> choose to create or edit an identity policy) does not include predefined action statements for Kafka-related objects.
• ORION-249325: When you create the first block view on the cluster via VAST CLI or VAST REST API, the VAST Web UI does not display the Element Store -> Volumes or Hosts tabs until you refresh the page.

VAST REST API

• In VAST Cluster 5.4, the response to a GET request sent to the /topics/ endpoint contains only the database name and topic name fields. It does not include other fields that were available in VAST Cluster 5.3.

Platform and Control

• The following limitations apply to conversion to write buffer RAID:

  • Conversion from VAST releases prior to 3.4 is not supported.

  • This capability is not supported for clusters with TLC drives, and also for VAST on Cloud clusters.

  • The cluster must include the following minimum number of DBoxes:

    DBox Type	DBox HA enabled	DBox HA disabled
    Ceres	15	4
    Mavericks	22	4

• The following rules and limitations apply when using Rack-Level Resiliency:
  • Every DBox must be associated with a single failure domain.
  • At least seven failure domains must be defined.
  • The number of DBoxes in the rack cannot exceed the number in any other rack by more than one.
  • The total available DBox capacity in a rack cannot be more than twice the available capacity in any other rack.
  • If two of the racks in a rack level-resilient cluster go down, bringing up only one of them does not return the cluster to normal operation. The cluster restores service only after the second of the racks goes up.
• Use of flash write buffers with DBox HA capability enabled requires at least 10 boxes.
• The following limitations apply to EBoxes:
  • ORION-193794: Power cycling of an EBox where the leader was running may result in significant IOPS degradation until the EBox is up again. Contact VAST Support for a workaround.
  • DBox migration is not available for EBoxes.

Callhome and Support

• An attempt to delete a support bundle before it is completely created may cause unexpected behavior.
• (RESOLVED IN 5.5.0) ORION-255750: The <bundle>.tar.progress file is not removed automatically when the bundle upload is finished.

Known Issues

Install and Upgrade

• ORION-270710: For some NVIDIA Mellanox NICs, the firmware upgrade process may take much longer than expected. Sometimes, a VMS timeout alert may be raised but the process still completes successfully. If a flint upgrade failure occurs during the process, try power-cycling the node.

Cluster Expansion

• ORION-220738: In some cases, VMS does not provide any alerts or other status indication when a drive gets disabled while the newly added DBox is being initialized.
• ORION-175762: In some cases, a DBox expansion procedure run on a cluster with similarity-based data reduction enabled can take longer than expected.

Multi-Tenancy

• ORION-279664: The names of customized analytic reports created by a tenant admin, can be seen by other tenant admins. This issue affects only the report names; the data within the reports are not accessible from other tenants.

Quotas

• ORION-387587: A 503 Forbidden error is returned when a tenant admin tries to create or edit user quotas on directories within the tenant, regardless of the permissions configured for the tenant admin manager user in the VMS.

Quality of Service (QoS)

• ORION-236122: Intense read workloads may impact performance on views controlled with a QoS policy with the prioritization flag set.
• ORION-139913: When applying a QoS policy to NFSv3 access, both data and metadata are taken into account in QoS limit calculations, while with NFSv4.1, only data are considered.
• ORION-137986: Enabling a QoS policy for a view on which a mixed (read and write) workload runs, can result in decreased performance for the workload.

NFSv4

• (RESOLVED IN 5.5.0) ORION-238708: In some cases, an NFSv4.1 client attempting to move files to the trash folder may get a permission denied error due to an issue that may cause the trash folder to use a more restricting policy than expected.

SMB

• ORION-402172: Sporadic SMB disconnects may be encountered during SMB server-side copy.
• ORION-395371: SMB clients may in some cases encounter intermittent access denied errors. The issue would typically occur in a multi-domain environment when a user is a member of multiple groups belonging to a domain that is different from the user's home domain, and such user performs a SMB operation permissions for which are granted through one of these groups.
• ORION-142968: If a quota is exceeded during the process of coping a file to the VAST cluster, the copying process is stopped with a misleading error message: A device attached to the system is not functioning.

S3

• ORION-378803: When responding to a POST upload request with no object key, the cluster returns error code 400 followed by InvalidObjectName instead of InvalidArgument.

Block

• ORION-375653: In some cases, live monitoring of block volumes may cause CNode containers to restart.

Attribute-Based Access Control (ABAC)

• ORION-196170: When a parent and child NFSv4.1 view both have same ABAC tags on them, an attempt to mount the child view may result in a permission denied error. If this occurs, try setting the ABAC tags for the machine account that the client uses to mount the view.

VAST Audit Log

• ORION-360698: VAST Audit Log records for presigned POST requests specify the request type as PUT_OBJECT.

VAST DataBase

• (RESOLVED IN 5.5.0) ORION-278671: The size displayed for a projection table in the VAST Database UI (DataBase -> *VAST Database -> navigate to the table) may deviate from the capacity used by the table according to the Element Store -> Views page. A discrepancy of up to 10% can be observed.

VAST DataEngine

• (RESOLVED IN 5.5.0) ORION-333926: If the cluster admin generates keys for a local user during the time when the user is logged in to the VAST DataEngine UI (using the old keys), the newly generated keys become valid only after the VMS key cache entry expires. This issue does not occur when the new keys are generated by the users themselves.
  (RESOLVED IN 5.5.0) ORION-292066: In VAST DataEngine UI, when trying to connect to a Kubernetes cluster from a VAST tenant that has upper-case letters in its name, the request fails with an Invalid value: <...>: a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character error.

Replication

• ORION-393875: An attempt to add a third member to a replication group may got stuck if the two destination peers in the group run different VAST releases, as follows:
  • One peer is on version 5.4 or later, while the other is on a version earlier than 5.4.
  • One peer is on version 5.1 or later, while the other is on a version earlier than 5.1.
• ORION-391617: Simultaneously creating S3 buckets on 2 clusters configured to run synchronous replication with Bucket Replication enabled, where the bucket paths are being synchronously replicated to the other cluster, may fail. Either create both buckets on one cluster, or wait until the first bucket creation completes before starting the second.
• ORION-140894: When attempting to delete a protected path from the destination peer after an ungraceful failover, a Failed to delete following streams or a similar error occurs. The workaround is to manually change the destination peer's role to STANDALONE and retry the deletion.

Global Access

• ORION-145307: Bulk permission updates are not supported for files and directories on satellite clusters.

Authentication and Authorization

• ORION-371895: In some cases, when using LDAP groups for authentication when logging in to the VMS, the login may fail if the user is a member of more than 440 groups.
• ORION-25479: Latin characters are not supported with LDAP. If you attempt to pass, for example, a username encoded with a Latin character set, the LDAP sanity check res: Invalid credentials error is returned.

VMS

• ORION-401264: The VAST Prometheus Exporter endpoint /api/prometheusmetrics/user_view may produce duplicate metric entries. If you encounter this issue, contact VAST Support for a workaround.
  ORION-332874: Cluster's attempts to send email notifications through SMTP may fail with a certificate verify failed: self-signed certificate (_ssl.c:1006) error. If you encounter this issue, contact VAST Support for a workaround.
• ORION-329549: In some cases after a cluster upgrade or expansion, the Analytics -> Data Flow and Analytics -> Top Actors pages of VAST Web UI may show the No data to display error instead of actual workload data. If you encounter this issue, contact VAST Support for a workaround.
• ORION-300508: Some metrics may not be available in VAST Prometheus Exporter. Missing metrics include (but may be not limited to) CNode and DNode hardware metrics or CPU temperature, utilization, memory percentage, PCI error count, retransmitted segments, total correctable/uncorrectable memory errors. If you encounter this issue, contact VAST Support for a workaround.
• (RESOLVED IN 5.5.0) ORION-293054: VAST Prometheus Exporter exports duplicate entries of vast_fan_metrics_hardware_rpm and vast_fan_active metrics for Ceres v2 DNode fans.
• ORION-292325: When creating a customized analytics report for block volumes or hosts, some metrics intended to reflect a maximum latency are set to a fixed value when there are no block volumes on the cluster.
• (RESOLVED IN 5.5.0) ORION-282603: User-made changes to the csi role (one of default administrative roles on the cluster) are not preserved during an upgrade.
• ORION-143717: On a cluster with CNode Port Affinity configured, there is no way to expose the VAST DNS IP on a specific port (left or right).
• ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.
• ORION-89570: In some cases, capacity analytics for subdirectories cannot be reported due to an internal timeout. This issue occurs when there is an extremely large number of subdirectories to be estimated.

VAST Web UI

• ORION-394425: In the Analytics -> Data Flow page, if the Sort field is set to BW, MD IOPS or Latency, selecting the Scan option in the By field causes the page to hang. Use the Scan option only when the Sort field is set to IOPS or ROW/s.
• ORION-390094: The Protocols dropdown in the DataBase -> VAST Audit Log page does not include an entry for STS, although the audit log can contain STS-related events.
• ORION-371787: The Default Gateway field in managed application settings (Data Engine -> Applications -> choose to create or edit an application -> Network tab -> Advanced pane) does not work as expected.
• (RESOLVED IN 5.5.0) ORION-355178: The ​Acknowledge Filtered​ button in the ​Alarms and Events​​ page does not acknowledge the selected alarms.
• ORION-308654: When using the Export selected rows as CSV option of the VAST Web UI, the resulting file may contain all of the rows, including those that were not selected for the export.
• ORION-294888: The Infrastructure pages of VAST Web UI may display [object Object] instead of the actual CPLD/UBM values for Dell Turin CNodes.
• (RESOLVED IN 5.5.0) ORION-260502: The Capacity Estimation page (Analytics -> Capacity), which is designed to display capacity for directories or, in case of a VAST Database, for its schema, sometimes also includes VAST Database tables in capacity estimations.
• ORION-239505: The VAST Web UI toggles for selecting operations to be logged in the syslog (Settings -> Notifications -> Syslog Setup) do not enforce logging of the selected operation types as expected. If you encounter this issue, contact VAST Support for a workaround.
• ORION-234835: Some VAST Web UI pages might not allow for proper filtering or sorting by column where value presentation differs from that in the VAST internal database.
• ORION-203189: The External Netmask field in cluster networking settings (Settings -> Configure Network) does not accept alphabetic characters.
• (RESOLVED IN 5.5.0) ORION-175189: When querying a local user using the Aggregated context, the Leading GID and Primary group SID fields in the User Details dialog have a value of -1 instead of an empty string.

VAST CLI

• ORION-322120: When displaying path names in VAST CLI, the names written in right-to-left languages may appear spelled from left to right.
• (RESOLVED IN 5.5.0) ORION-269350: An illegal argument error occurs when trying to run the protectedpath list command with the --protection-policy-name option specified.
• ORION-265720: VAST CLI auto-completion does not include the --detach-krb-provider option on the tenant modify command.
• (RESOLVED IN 5.5.0) ORION-156628: An attempt to run the viewpolicy show --audit command results in a 'ViewPolicyProtocolsAudit' object has no attribute 'get' error.

VAST REST API

• ORION-178569: The /users/names endpoint always returns only the first 50 entries, regardless of the page size parameter or the total amount of entries to be returned.

Platform and Control

• ORION-403644: If an administrative role has OS SSH login enabled (Administrators -> Administrative Roles -> select a role -> Enable OS SSH Login option), an attempt to enable OS SSH login for a second role may result in an internal timeout, which also causes the feature to stop working for the first role.
• ORION-275610: The Drive Compatibility page in VAST Web UI (Support -> Drive Compatibility) does not display all of the details that are available when running a supporteddrives list or supporteddrives get command of VAST CLI.
• ORION-255054: Once a DNode replacement operation is complete, the old node may still be listed in the Infrastructure -> DNodes page of VAST Web UI for some minutes.

Callhome and Support

• ORION-239170: When obfuscating a support bundle, the CNode hostname may not get obfuscated in some of the logs included in the bundle.