VAST Cluster 5.4.0 Release Notes

To access this release, contact the VAST Customer Success team via a support ticket, Slack, or by sending an email to support@vastdata.com.

Upgrade to VAST Cluster 5.4.0 is supported from the following releases:

• 5.3.3 - 5.3.3-SP4
• 5.3.2 - 5.3.2-SP7
• 5.3.1 - 5.3.1-SP3
• 5.3.0 - 5.3.0-SP8

Note that direct upgrade may not be supported from hotfix builds. Consult VAST Support regarding upgrade if your cluster is running a hotfix build.

• New Features
• Enhancements
• Resolved Issues
• Limitations
• Known Issues

For scale-related information, see VAST Cluster Scale Guidelines.

New Features

• NETWORKING

  • L3 Numbered BGP

• ENCRYPTION OF DATA OVER THE WIRE

  • Support for Kerberos Providers other than Active Directory Kerberos

• PROTOCOLS

  • NFSv4 File Delegations

  • Support for SMB Encryption

  • S3 Server-Side Encryption with Customer-Provided Keys (SSE-C)

• VAST EVENT BROKER

  • Kafka Authentication and Authorization

  • Support for Kafka Config APIs

  • Kafka Topic Compaction

  • Publishing S3 Notifications to VAST Event Broker

  • Data Protection Capabilities for Event Topics

• VAST DATABASE

  • VAST Query Engine

  • VAST Database Row/Column-Level Security

  • Sorted Tables in VAST Database

  • Support of Vector Operations

• VAST DATA ENGINE

  • Running Trino Clusters on VAST

  • Provisioning for Deployment of User Functions and Pipelines

• DATA PROTECTION

  • Replication and Global Access on the Same Path

• AUTHENTICATION & AUTHORIZATION

  • Security Token Service (STS), IAM Roles and OIDC Providers

• VMS

  • Monitoring of Data Pending Deletion and Total Snapshots Capacity

• SUPPORT

  • YAML-Based Drive Compatibility Validation

L3 Numbered BGP

VAST cluster L3 networking capabilities have been expanded by adding support for Numbered BGP, which relies on peer IP assignments to establish peer-to-peer relationships.

In VAST Web UI, BGP settings (Network Access -> BGP Configurations -> choose to create or edit a BGP configuration) have been updated to let you:

• Select numbered or unnumbered BGP.

• In the new VIPs tab:

  • Enter a range of virtual IP addresses to be used for peer-to-peer connections.

  • Set whether the CNodes/switch ports are assigned Even or Odd IP addresses from the specified range.

In addition, the BGP Connections page in VAST Web UI has been renamed to BGP Peering and includes indication of the source IP and peer IP for each BGP connection.

The following limitation applies:

• ORION-266297: When using numbered BGP, each CNode can only be involved in one BGP configuration. To learn which CNodes are currently used in which BGP configurations, use the Network Access -> Virtual IP Pools page that displays CNodes and BGP Configurations for each virtual IP pool.

Support for Kerberos Providers other than Active Directory Kerberos

VAST Cluster 5.4 expands Kerberos support so that VAST cluster tenants can use MIT Kerberos as an authentication provider.

The following user controls have been added for this purpose:

• In VAST Web UI, the User Management -> Providers page includes tabs for all providers that you can associate with a VAST cluster tenant. To create a new Kerberos provider, open the Add New Provider dropdown on the top right of the page and select Kerberos.

  After you've added a Kerberos provider, the User Management -> Providers page will include a new tab named Kerberos designed to let you view and manage Kerberos providers.

• In VAST CLI:

  • Use the kerberos * commands to create and manage Kerberos providers.

  • To associate a Kerberos provider with the tenant, specify the --krb-provider-id option on the tenant create and tenant modify commands.

  • To remove an association between a Kerberos provider and a tenant, specify the --detach-krb-provider option on the tenant modify command.

The following rules and limitations apply:

• MIT Kerberos providers are used for NFSv4 only.

• A tenant can be associated with a single Kerberos provider only.

• Up to eight Kerberos providers can be defined on a VAST cluster.

• Kerberos principals used to authenticate to the VAST cluster must exists as users in an LDAP provider or in a VAST provider associated with the same tenant as the Kerberos provider. Otherwise, they are squashed to the nfsnobody user.

The following known issue may be encountered:

• ORION-265720: VAST CLI auto-completion does not include the --detach-krb-provider option on the tenant modify command.

NFSv4 File Delegations

The VAST NFSv4 server can optionally grant a read or write delegation to the client opening a file.

• If the server grants a read delegation, the client can cache the data and read from the cache, while the server assures that no other client can write to the file for the duration of the delegation.

• If the server grants a write delegation, the client can cache data and metadata, and perform both read and write operations on it, while the server assures that no other client can write to the file or read from it for the duration of the delegation.

Use of file delegations may significantly reduce the amount of client-server communication for the files being delegated, boosting VAST cluster performance for specific types of workloads, such as cloning or getting status information of a GitHub repository, running SQLite queries, compilation tasks, or extracting data from very large ZIP or TAR archives.

By default, on new installations, NFSv4 file delegations are enabled for both read and write operations. In upgraded environments, the pre-existing tenants have NFSv4 file delegations disabled.

The following user controls have been added for this feature:

• In VAST Web UI:

  • The Enable read delegations and Enable write delegations toggles, and also the Enable Unrequested NFSv4 File Delegations by Default in tenant settings (Element Store -> Tenants -> choose to create or edit a tenant -> Advanced Protocol Settings tag)

  • The page for view's locks and open handles has been expanded to let you list and manage existing NFSv4 delegations per view (Element Store -> Views -> right-click a view and select Locks -> File Byte-range locks and open handles and file delegations).

• In VAST CLI:

  • The --allowed-delegations, --enable-grant-unrequested-delegations-by-default and --disable-grant-unrequested-delegations-by-default options on the tenant create and tenant modify commands

  • The tenant nfs4-delegs-list and tenant nfs4-delegs-remove commands

The following requirements and limitations apply when using NFSv4 file delegations:

• The NFSv4 protocol requires Network Time Protocol (NTP) to be configured on both the VAST cluster and the client. While VAST does not enforce this requirement, the absence of proper time synchronization may, in rare cases, result in unexpected behavior.

• The client must have an NFSv4 backchannel connection.

• NFSv4 delegations are not supported with RDMA.

• For multi-protocol views with NFSv4 delegations enabled, close-to-open consistency is only preserved between NFSv4 clients, but not between NFSv4 clients and other protocols (including NFSv3). For example, if a non-NFSv4 client modifies a file on such a view, the modification may not become immediately visible to an NFSv4 client holding an active delegation on the file.

• Before making a path read-only by using the /folders/read_only endpoint of VAST REST API, ensure that NFSv4 delegations are disabled or returned. Otherwise, unexpected behavior may be encountered.

• Restoring a snapshot on a tenant that have NFSv4 delegations enabled, may cause unexpected behavior with regard to delegated files. It is recommended to revoke existing delegations before restoring a snapshot.

• VAST Cluster recalls granted NFSv4 delegations when a failover process is started.

• In some cases when there is very high NFSv4 callback load per cluster or per client, VAST Cluster might start recalling or revoking NFSv4 delegations due to callback request timeouts.

• NFSv4 delegations require mounting the client at least as NFS version 4.1. It is recommended to mount as NFS version 4.2 for additional performance improvements. Linux kernels 6.11+ on a client provide more performance advantages (RFC9754), which get backported to older kernels through VAST-NFS.

• If you are using VAST-NFS, note that NFSv4 delegations are supported with VAST-NFS 4.0.32 and later. VAST-NFS 4.5 includes enhancements related to support of NFSv4 delegations.

• SUSE Linux Enterprise Server 12.x and CentOS 7.x clients are not supported with NFSv4 delegations enabled.

For more information, see VAST Cluster Administrator’s Guide.

Support for SMB Encryption

VAST Cluster 5.4 supports SMB 3.0 encryption, which helps protect in-flight data on non-secure networks.

By default, SMB encryption is disabled. You can enable it and select an encryption policy per tenant:

• Available - Encryption is used only for SMB clients which have requested it explicitly. For clients that do not support encryption, access is allowed but no encryption is used.

• Desired - The cluster uses encryption for any SMB client that supports encryption. For clients that do not support encryption, access is allowed but no encryption is used.

• Required - SMB clients that do not support encryption are denied access.

If the tenant has SMB encryption enabled with one of the encryption policies set, you can override the tenant's setting by choosing a different encryption policy for a view.

A view cannot have a less strict encryption policy compared to that of the tenant. For example, if the tenant has Desired, the view can have Required but not Available.

An access denied error is returned if the cluster configuration stipulates use of encryption but the client does not use SMB 3.0, does not support encryption, or sends a non-encrypted packet.

Note: Enabling SMB encryption may result in up to 15% performance degradation compared to SMB signing.

To configure SMB encryption for a tenant:

• In VAST Web UI, go to the Advanced Protocol Settings tab in tenant settings (Element Store -> Tenants -> choose to create or edit a tenant) and in the SMB Encryption pane, toggle the Enable encryption option on and select an encryption policy.

• In VAST CLI, run the tenant create or tenant modify command with the --smb-encryption-state option specified.

To configure SMB encryption for a view:

• In VAST Web UI, go to the SMB -> Encryption tab in view settings (Element Store -> Views -> choose to create or edit a view) and in the Set Activation Policy pane, select an encryption policy.

• In VAST CLI, run the view create or view modify command with the --smb-encryption-state option specified.

Note: The view must have the SMB protocol enabled.

For more information, see VAST Cluster Administrator’s Guide.

S3 Server-Side Encryption with Customer-Provided Keys (SSE-C)

VAST Cluster can perform server-side encryption (SSE) of S3 data at rest with the encryption keys provided by the client (SSE-C).

With SSE-C, the client provides and manages an encryption key used to encrypt the S3 objects at their destination. The cluster holds the key only for the time needed to handle the request and does not store it persistently. When accessing an encrypted object, the client is expected to provide the same encryption key as the one used to encrypt the object.

The following operations may involve SSE-C encryption:

• GetObject

• HeadObject

• PutObject

• CreateMultipartUpload

• UploadPart

• CopyObject

• CopyPart

• UploadPartCopy

• Presigned POST

The client is expected to use the x-amz-server-side-encryption-customer-* headers to pass an 256-bit, base64-encoded encryption key, the encryption algorithm to be used and the unencrypted key checksum with each request that requires encryption of the object at its destination. For each CreateMultipartUpload request, all the subsequent UploadPart requests must contain the same values in the headers as the CreateMultipartUpload request.

Support for S3 SSE-C is enabled by default. No manual configuration is needed.

The following rules and limitations apply:

• If an S3 request contains an x-amz-server-side-encryption-customer-* header, it is required to use HTTPS for the request to be accepted by the cluster.

• Only AES256 encryption algorithm is supported.

• Objects encrypted with SSE-C cannot be accessed through other access protocols.

• For replication environments, it is strongly recommended to avoid using SSE-C unless all of the clusters involved support SSE-C.

• S3 SSE-C is not supported when uploading objects using chunked encoding.

The following known issue can be encountered:

• (RESOLVED IN 5.4.1-SP4) ORION-321170: Using S3 SSE-C when uploading objects using chunked encoding may result in unexpected errors.

For more information, see VAST Cluster Administrator’s Guide.

Publishing S3 Notifications to VAST Event Broker

With VAST Cluster 5.4.0, you can publish events occurring in the S3 Bucket views for which S3 event notifications are configured, to one of the Kafka-enabled views on the same VAST cluster.

To do so in VAST Web UI, select the destination Kafka-enabled view in the Broker field of event notification settings (Element Store -> Views -> choose to create or edit a view -> S3 -> Bucket Notifications tab).

Kafka Topic Compaction

VAST Cluster 5.4.0 supports Kafka topic compaction.

If compaction is enabled for a topic, any old records get deleted when there is a newer version of the record with the same key in the partition log. Compaction is performed asynchronously in the background, meaning that at some point in time, duplicate keys may exist.

To enable or disable topic compaction in VAST Web UI, use the new Compaction flag in topic settings (DataBase -> VAST Database -> navigate to the topic you need and open its properties).

Support for Kafka Config APIs

VAST Cluster 5.4 supports describeConfigs and alterConfigs Kafka APIs that let you get and update the configuration for event topics.

You can view and change the configuration parameters in VAST Web UI, CLI and via REST API.

The topic settings dialog in VAST Web UI has been expanded to include more configuration parameters, and also to support the edit mode.

Kafka Authentication and Authorization

This release introduces support for Kafka-compatible authentication of users connecting to Kafka-enabled views on the VAST cluster.

The VAST cluster supports SASL plain authentication by username and password on both encrypted and non-encrypted connections.

VAST implementation of user authorization is based on identity policies rather than on Kafka ACLs. In identity policies, you can list permissions for various Kafka operations against topics, consumer groups, and the Kafka cluster.

To set up Kafka authorization and authentication:

• In VAST Web UI, use the Authentication Methods pane in Kafka-enabled view settings (Element Store -> Views -> choose to create or edit a view -> Kafka tab)

• In VAST CLI, run the view create or view modify command with the corresponding options specified:

  • Authentication on encrypted connections: --enable-kafka-encrypted-conn, --kafka-encrypted-auth-mechanism, --disable-kafka-encrypted-conn

  • Authentication on non-encrypted connections: --enable-kafka-unencrypted-conn, --kafka-unencrypted-auth-mechanism, --disable-kafka-unencrypted-conn

  • Authorization: --require-kafka-authorization or --cancel-kafka-authorization

The following limitations applies:

• Active Directory/LDAP is not supported. The user must be defined as a VAST local user.

• Only one Kafka TLS certificate can be uploaded per VAST cluster.

For more information, see VAST Cluster Administrator’s Guide.

Data Protection Capabilities for Event Topics

The following data protection capabilities are supported for VAST Event Broker topics:

• VAST asynchronous replication and failover

  The replicated topics are read-only at the destination peer. In case of a failover, the destination peer becomes the one hosting the topics to which events are produced.

• Snapshots and global snapshot cloning

  A clone can be created from either local or remote snapshots.

The following rules and limitations apply:

• The Kafka-enabled view needs to be manually created and associated with a virtual IP pool at the destination peer. The pool must have the same name as the one at the source peer.

• Fast restore of a protected path containing a Kafka-enabled view is not allowed.

• Consumer group offsets are not replicated.

Support of Vector Operations

VAST Cluster 5.4 adds support for the vector data type (arrays of floats), enabling users to run similarity queries against a VAST database. Vector operations are supported with both VAST Query Engine and VAST Python SDK. For more information and examples, see VAST Cluster Administrator's Guide.

The following limitation applies:

• VAST Cluster 5.4.0 does not include performance optimization for vector operations.

Sorted Tables in VAST Database

VAST Cluster 5.4 adds support for sorted database tables. Sorted tables drastically improve performance for selective queries. After sorting has been enabled for a table, a background task keeps it sorted as rows are being added and deleted. Queries against the table can be run during the sorting.

Sorting can be enabled via VMS and from the Trino query engine or VAST DB SDK. To enable sorting on a table and set the sorted columns via VMS:

• In VAST Web UI, for a new table, use the new Enable Table Sorting option in the Add Table dialog (VAST DB -> VAST Database -> browse to the schema -> choose to create a new table). Alternatively, if the table has already been created, navigate to it and select Enable Table Sorting in its action menu.

• In VAST CLI, list the sorted columns on the --sorted-column-names option of the table create or table modify command.

To view sorted columns in a table:

• In VAST Web UI, navigate to the table from the VAST Database page and open its properties.

• In VAST CLI, use the --list-sorted-columns option on the column list command.

For each sorted table, a sorting status is provided, which is determined based on the amount of rows already sorted. A table is guaranteed to reach the sorting score of Sorted. It may reach the status of Super sorted depending on how pre-ordered the existing table data is.

The following limitations apply:

• Once enabled for a table, the sorting cannot be disabled.

• Sorting can be performed for tables that contain 512k or more rows. Although you can enable sorting on a table regardless of its size, smaller tables do not get sorted.

• Once sorted, tables cannot be unsorted or have their sorted columns changed.

• Only one sorting order can be defined on a table. Up to four sorted columns are supported.

• Sorted column values cannot be updated to different values.

• Sorted tables cannot be replicated. Tables that are replicated cannot be sorted.

• Sorting cannot be enabled on tables that have semi-sorted projections. Semi-sorted projections cannot be added to sorted tables.

• Nested data types are not supported for sorted columns.

• Tables that expose the internal row ID with the vastdb_rowid column cannot be sorted.

• When calculating the sorting status, a constant value within the Sorted value range is returned for tables that contain less than 5 million rows.

Warning: Some background tasks related to processing of sorted tables might transiently consume space beyond quota limits, resulting in quota exceeded alerts for the users.

For more information, see VAST Cluster Administrator’s Guide.

VAST Database Row/Column-Level Security

You can control access to information stored in a VAST database table by defining the permissions at the row or column level.

The row or column permissions are specified in an identity policy or a bucket policy. The policies must use a new format for the VAST DB row and column permission statements, where resource is a VAST database table or view.

To build an identity policy with this type of statements, open the VAST Web UI policy editor (User Management -> Identity Policies -> choose to create or edit a policy) and select Database Row/Column Security in the Define Statements pane, then specify the resources to be included or excluded, and set row filters and column masks as needed.

The following requirements and limitations apply:

• Row filtering and column masking are supported for Trino query engines only.

• The Trino user's identity policy must have the EndUserImpersonation and GetRowColumnSecurity actions allowed.

For more information about this feature, see VAST Cluster Administrator's Guide.

VAST Query Engine

VAST Cluster 5.4.0 includes VAST implementation of a database query engine, the VAST Query Engine, that boosts performance for database operations.

Among capabilities supported by VAST Query Engine in release 5.4.0 are vector search, row/column-level security, and filter pushdown.

VAST Database clients can access the query engine using the VAST ADBC driver that connects to a dedicated virtual IP pool using a preconfigured S3 user account. The virtual IP pool must be assigned the new Query Engine role.

For more information about VAST Query Engine and VAST ADBC driver, see VAST Cluster Administrator's Guide.

Running Trino Clusters on VAST

In addition to running managed Spark applications, you can configure and run a Trino cluster on your VAST cluster.

To set up a Trino cluster, you select the CNodes, set the limit for CNode resource utilization, configure networking and upload a YAML configuration file that defines all the necessary Trino settings. A template YAML file is provided for your convenience. If you want to secure the connection with TLS, you can also upload the TLS certificate and the keys.

You can run Trino and Spark applications concurrently on the same VAST cluster. In this case, a separate set of CNodes is required for each of the applications.

VAST Web UI has been updated to let you select the Trino application type in the Application field of managed application settings (Data Engine -> Applications -> choose to create or edit an application).

The following rules and limitations apply:

• Trino 475 is supported.

• Not less than three CNodes are required per Trino cluster (one for the coordinator, two or more for the workers).

• In case of CNode HA events, the Trino cluster remains accessible and running queries as long as one coordinator CNode and one of the worker CNodes are up.

• A VAST cluster upgrade (NDU) performed while running a Trino cluster may cause Trino query failures. The Trino cluster will be fully operational after the NDU is complete.

For more information, see VAST Cluster Administrator’s Guide.

Provisioning for Deployment of User Functions and Pipelines

VAST DataEngine offers VAST cluster tenant users an ability to create and run functions and pipelines triggered by certain types of events or schedules. A user can write application code in Python and host it on Kubernetes, letting the VAST cluster manage all infrastructure aspects of the deployment.

To make the process as simple as possible, VAST provides a new dedicated web-based UI, CLI and REST API for VAST DataEngine users to write and manage their application code. The new interfaces are only available for authorized users of a tenant for which VAST DataEngine provisioning has been enabled, and can be used to create, modify, delete and view functions, pipelines and triggers, and also to deploy and run functions and pipelines.

A set of built-in functions is available which can be used as building blocks to model use cases.

Deployment of VAST DataEngine involves setting up a dedicated VAST Event Broker view or third-party event broker (to stream events to), and also connecting the VAST cluster to a container registry (to store images of functions to be deployed) and a Kubernetes cluster (to consume the events and run the functions on them).

For more information about VAST DataEngine, see VAST DataEngine User Guide and VAST Cluster Administrator's Guide.

In addition to the new dedicated interfaces, the following user controls have been added for this feature in VMS:

• To enable DataEngine for a tenant:

  • In VAST Web UI, the new Enable DataEngine option in the context menu for a tenant listed in the Element Store -> Tenants page. This option starts a wizard that lets you enable the functionality for the tenant, assign a VAST Event Broker view, connect to a Kubernetes cluster and a container registry.

• To configure a dedicated virtual IP pool for DataEngine:

  • In VAST Web UI, go to virtual IP pool settings (Network Access -> Virtual IP Pools -> choose to create a pool -> General pane) and select Query Engine in the Role field.

• To manage Kubernetes clusters and container registries, use the Data Engine -> Kubernetes clusters and Data Engine -> Container Registry pages in VAST Web UI.

• To set up application user group and permissions:

  • In VAST Web UI:

    • When adding an administrative role (Administrators -> Administrative Roles -> choose to create or edit a role), you can select the Tenant admin / Application user option to indicate that the role will be used to grant permissions for application users.

    • The Users group field in the Who Can Access This Tenant (Data Engine) pane of User Access Management tab in tenant settings (Element Store -> Tenants -> choose to edit a tenant) lets you specify the application user group.

    • Identity policy settings include a new field named Resource type where you can select Data Engine and proceed to entering statements that are applicable for DataEngine.

  • In VAST CLI, use the following options on the tenant create and tenant modify commands:

    • --application-users-group-name

    • --enable-data-engine-role and --disable-data-engine-role

    • --enable-data-engine-s3-policy and --disable-data-engine-s3-policy

The following limitations apply:

• If VAST DataEngine is enabled on the cluster, replication of the tenant's root directory is not allowed.

• Up to 32 engines are supported per VAST cluster.

• (RESOLVED IN 5.4.1) ORION-288279: VAST DataEngine does not support timestamps with a time zone.

• (RESOLVED IN 5.4.1-SP5) ORION-302749: When using an external Kubernetes cluster, it is recommended to implement the following node-level configuration to prevent the pods from occasionally failing due to the ​Too many open files​​ error:

  1. Configure ​containerd​​/​docker​​ on all worker nodes to increase the default file descriptor limit:

     # /etc/systemd/system/containerd.service.d/override.conf
     [Service]
     LimitNOFILE=1048576:1048576 
     

  2. Run ​systemctl daemon-reload && systemctl restart containerd​​.

Known issues are as follows:

• ORION-333926: If the cluster admin generates keys for a local user during the time when the user is logged in to the VAST DataEngine UI (using the old keys), the newly generated keys become valid only after the VMS key cache entry expires. This issue does not occur when the new keys are generated by the users themselves.

• ORION-293900: When streaming S3 event notifications or VAST DataEngine function-triggering events to a VAST Event Broker view on a cluster with multiple VAST Event Broker views belonging to different tenants, some of the event notifications may encounter a flow that prevents them from being sent to their destination. If this occurs, an Internal Kafka target supported only a single tenant GUID, got 2 alert will be raised. With VAST DataEngine enabled, this issue may result in some functions not being triggered as expected.

• (RESOLVED IN 5.5.0) ORION-292066: In VAST DataEngine UI, when trying to connect to a Kubernetes cluster from a VAST tenant that has upper-case letters in its name, the request fails with an Invalid value: <...>: a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character error.

• ORION-288909: When trying to list logs or traces in VAST DataEngine CLI, the --tenant option on the CLI command does not work as expected.

Replication and Global Access on the Same Path

VAST Cluster 5.4.0 adds support for a configuration where the protected path which is the source for asynchronous replication, is also configured as a Global Access source path, while the replication destination path is used as the Global Access destination path. (Prior to this release, use of the same destination cluster was not supported.)

This configuration requires that asynchronous replication is enabled first, and then the Global Access configuration is enabled.

The following user controls have been added for this purpose:

• In VAST Web UI:

  • The Activate global access and Activate on source as well flags in the destination settings of a remote protected path (Data Protection -> Protected Paths -> choose to create a path -> open destination settings)

  • The Replication and Global Access data service type in VAST Data Space (VAST Data Space -> choose to add a new data service)

• In VAST CLI:

  • The --source-member-capabilities option on the protectedpath create, protectedpath add-stream, protectedppath modify-member commands

Security Token Service (STS), IAM Roles and OIDC Providers

VAST Cluster lets you provide users with temporary S3 access keys to enable short-term access to S3 buckets based on a security token sent to the new VAST Security Token Service (STS).

The user sends an assume role request to the VAST STS. The request contains the username, the IAM role to be assumed, and a JSON Web token (JWT) (retrieved from the authentication provider). VAST STS validates the JWT and responds with a temporary S3 access key that provide access permissions according to the assumed role.

You need to define the IAM roles to be assumed on the VAST cluster. An IAM role definition includes associated identity policies (optionally, to specify permissions that this role will grant) and a trust policy. A trust policy determines if a specific JWT is allowed or rejected.

The following user controls have been added for IAM roles:

• To view and manage IAM roles on the VAST cluster:

  • In VAST GUI, the new User Management -> IAM Roles page

  • In VAST CLI, the iamrole * commands.

• To specify that the owner of an S3 bucket is a IAM role:

  • In VAST Web UI, the User and IAM Role toggles in S3 general settings of a view (Element Store -> Views -> choose to create or edit a view -> S3 tab -> General pane)

  • In VAST CLI, the --bucket-owner-type option on the view create and view modify commands.

• To view and manage the association between a IAM role and a user QoS policy:

  • In VAST Web UI, the IAM Role option in the Users or IAM Roles dropdown when editing a user QoS policy (Element Store -> QoS policies -> choose to create or edit a QoS policy -> select the User policy type and go to the User tab)

  • In VAST CLI, the --attached-iam-roles option on the qospolicy create, qospolicy modify, qospolicy show commands.

• To view IAM roles for a user quota:

  • In VAST Web UI, the IAM Role Accounting tab when editing a user quota (Element Store -> Quotas -> open a user quota)

  • In VAST CLI, the --iam-role-rules option on the quota show command.

Users are authenticated by a third-party OpenID Connect (OIDC) provider, which issues a JWT with which the user requests to assume a role. To create and manage OIDC providers on the the VAST cluster:

• In VAST Web UI, go to User Management -> VAST Providers page, open the Add New Provider dropdown and select OIDC from the list.

• In VAST CLI, use the oidc * commands.

To manage the association between an OIDC provider an a VAST tenant:

• In VAST Web UI, use the Providers and User Access -> Set Providers -> OIDC tab in tenant settings (Element Store -> Tenants -> choose to create or edit a tenant).

• In VAST CLI, to create the association, specify the --oidc-provider-id option on the tenant create or tenant modify command. To remove the association, use the --detach-oidc-provider option on the tenant modify command.

STS operations are subject to VAST audit logging. To enable logging of STS operations:

• In VAST Web UI, select STS in the protocol selection field in global audit settings (Settings -> Auditing -> Global Baseline Audit Settings -> Select protocols to assign operations).

• In VAST CLI, specify the STS value on the --audit-protocols option of the cluster modify command.

The following restrictions and limitations apply:

• STS is supported with HTTPS only.

Use of STS requires that the S3 protocol is enabled for the cluster.

• One OIDC provider per VAST cluster tenant.

• When replicating IAM roles:

  • ORION-278747: IAM role replication may take several minutes to complete in case the cluster is under high load or there is a very large amount of roles to replicate.

  • STS access keys are replicated only when the protection path is in SYNC state.

  • ORION-278217: When processing the assume role operation, the cluster may be raising a Failed to replicate STS temp keys alert until the synchronous replication stream/protection path reaches the SYNC state.

• ORION-285727: In some rare cases when IAM roles are subject to replication, if you delete an IAM role on a replication source, the deleted role gets recreated as a result of the replication. The recreation might take place, for example, in case a HA event occurs during role deletion, or if you modify the role on the replication destination (at the same time as the role is being deleted on the replication source). If you encounter this issue, delete the recreated role again both on the replication source and destination.

• ORION-287563: Only five OIDC keys can be returned for a manual request to get or refresh OIDC keys.

For more information, see VAST Cluster Administrator’s Guide.

Monitoring of Data Pending Deletion and Total Snapshots Capacity

VAST Cluster offers an ability to monitor two new metrics, which in sum represent the cluster's auxiliary capacity:

• How much data is pending deletion on the cluster, i.e. expected to be deleted by VAST asynchronous processes without further user interaction (such as data in the Trash folder).

• How much capacity is used to store snapshots, i.e. how much space will be freed if all of the snapshots are deleted.

The new metrics are shown on the cluster's dashboard as Pending Deletion and Snapshots usable and logical capacities, replacing the deprecated Auxiliary capacity metric in the Capacity widget.

The Cluster Auxiliary Capacity analytics report has been enhanced to include both new metrics.

For more information, see VAST Cluster Administrator’s Guide.

YAML-Based Drive Compatibility Validation

When adding a new drive to the cluster as part of the cluster expansion or field replacement procedure, the cluster checks the drive against the list of supported drives and raises an alarm if the new drive is not found on the list. Non-supported drives are rejected.

Drive compatibility information is maintained using a YAML file (provided and signed by VAST Support) that contains detailed information for the supported drives. This eliminates the need to run a cluster upgrade each time new drives are qualified to be used in VAST clusters.

YAML-based drive compatibility validation is enabled by default. If you want to have it disabled so that drives are verified against a hard-coded list initially supplied with the product, contact VAST Support.

The following user controls have been added for this feature:

• In VAST Web UI, the new new Support -> Drive Compatibility page in VAST Web UI lists drives that are compatible with the current VAST Cluster version. For each drive, the page indicates the drive model name and number, supported firmware versions, capacity and role (boot drive, SSD, SCM).

• In VAST CLI, the supporteddrives list and supporteddrives get commands

The following known issue was encountered for this feature:

• ORION-275610: The Drive Compatibility page in VAST Web UI (Support -> Drive Compatibility) does not display all of the details that are available when running a supporteddrives list or supporteddrives get command of VAST CLI.

Enhancements

Install and Upgrade

• Added an ability to skip failed CNodes during a VAST Cluster Install procedure. The skipped CNodes can later be added through cluster expansion.

  By default, CNodes can be skipped only when the number of failed CNode does not exceed 25% of the total number of CNodes in the cluster. Otherwise, installation cannot be completed.

  The skipping threshold can be adjusted in VAST Cluster Install. If necessary, you can also disable the feature.

• ORION-237652: Added an ability to specify the NTP server hostname when supplying NTP parameters in VAST Cluster Install.

• ORION-232534: Added support for specifying zero-padded IPv6 addresses during cluster deployment. With this enhancement, both 2001:db8:e:14:0:0:0:ff and 2001:db8:e:14::ff formats are supported.

Cluster Expansion

• ORION-216963: Added a checkpoint mechanism for the cluster expansion procedure. Checkpointing splits the procedure into a series of discrete steps. If one of the steps fails, the process can be resumed from the failed step, without the need to rerun the steps that have already completed.

Networking

• ORION-242967: Enhanced the cluster networking configuration script (configure_network.py) to support configuring CNode Port Affinity for Supermicro Gen5 CNodes and HPE Genoa CNodes.
• ORION-237376: Added a validation to prevent users from adding VAST reserved IPs (e.g. IPs that are already in use by the cluster, such as the VMS IP) to a virtual IP pool.
• ORION-203793: Added indication of the port type (internal, external, management) when listing NICs.

Encryption

• EKM-based encryption can be enabled not only during cluster deployment, but also afterwards, when the cluster is already up and running.

  • In VAST Web UI, the Data Management and KMIP tabs in cluster settings (Settings -> Cluster) has been redesigned to let you select the required encryption type and configure third-party EKM servers on a running cluster.

  • In VAST CLI, you can manage EKM servers and certificates using the new cluster add-ekm and cluster set-certificates commands.

  Note: Tenants created before enabling EKM on the cluster, will be assigned an Internal encryption group, which means no EKM will be used for such tenants.

Multi-Tenancy

• ORION-276035: Increased the maximum allowed number of roles per tenant from 1000 to 4500.
• ORION-211489: Added an ability to set a tenant-wide limit on the number of views that can be created for the tenant. If left unlimited (which is the default), the number of views will only be capped by the maximum amount of views supported by the VAST cluster.

To set the limit for a tenant:

• In VAST Web UI, use the new Max Number of Views pane in advanced tenant settings (Element Store -> Tenants -> choose to create or edit a tenant -> Advanced Protocol Settings tab).

• In VAST CLI, use the --max-views option on the tenant create or tenant modify command.

• ORION-173326: Increased the maximum allowed number of tenants per cluster from 4096 to 10240.

Quotas

• Increased the maximum allowed number of nested quotas from 3 to 16.

Quality of Service (QoS)

• ORION-243368: Added support for prioritization of workloads based on the QoS policy prioritization flag in scenarios where the policy also have burst and/or total limits defined.

SMB

• ORION-196158: Added support for the SMB2_CREATE_APP_INSTANCE_ID create context for directories. Prior to this change, it was supported for files only.

S3

• ORION-208315: Added support for checksum validation of PUT requests using the Content-MD5 header. Prior to this change, the header was ignored.

  Note: Lifecycle rules are not supported.

Block

• The volume list command of VAST CLI has been enhanced to provide various output filters and display options.
• The view list command of VAST CLI offers two new output filters, --is-default-subsystem and --is-not-default-subsystem, that let you show or hide block views set as default subsystems.

Event Publishing

• The new --kafka option on the VAST CLI view show command displays Kafka protocol settings in effect for the view.
• ORION-253455: Added an ability to specify a value of -1 for the topic retention period. Setting -1 means no retention occurs.

VAST DataBase

• Added an ability to modify properties of an existing table in VAST CLI using the new table modify command.
• Added support for TZ (timestamps) and UUID data types.
• Added support for over-the-wire compression of query results. The compression can be enabled for a Trino session by using the compression session parameter set to zstd. By default, compression is disabled.

Replication

• Replication is no longer limited to a maximum of two destination peers when VAST Catalog is enabled, as long as all of the clusters involved run VAST Cluster 5.4 or later.
• It is no longer required to manually disable VAST Catalog before performing operations that require a reset to existing snapshots, such as replication failover, disconnection/reconnection of one of the replication peers, switching of the peer roles, or deletion of a protected path.
• ORION-95871: Added an ability to configure the cluster to allow or prohibit deletion of an empty directory for which one or more snapshots exist. By default, deletion of such a directory is not allowed. If you want to alter the default behavior and allow deletion of such directories on your cluster, contact VAST Support.
• ORION-283592: VAST Cluster can be configured to ignore duplicate entries that might exist for a user or group in the same LDAP domain. If, for example, two users share the same UID, one of these user entries will be ignored, thus ensuring consistent outcome of access checks for that UID. To enable this behavior on your VAST cluster, contact VAST Support.
• ORION-281740: Added an ability to configure the cluster so that replicated identity or bucket policies are automatically enabled on the replication destination peer. By default, the replicated policies are kept disabled, and manual action is required to enable them. If you want to alter this behavior so that the replicated policies are enabled automatically, contact VAST Support.

Authentication and Authorization

• Added an ability to optionally set a password for a VAST local user account.

  The cluster or tenant admin generates a temporary password which the user must enter on their first login to the VMS. Upon logging in, the user is prompted to set a permanent password.

  The user password must meet the requirements set in VMS settings (in VAST Web UI: Settings -> VMS -> Password).

The following user controls have been added for this purpose:

• In VAST Web UI, the User Password pane in local user settings (User Management -> Local users -> choose to create or edit a user)

• In VAST CLI, the --password option on the user create and user modify commands.

VMS

• ORION-236019: Updated VAST Prometheus metrics to add a metric that indicates the total number of all open NFS connections (including NFS3, NFS4, RQUOTA, MOUNT, NLM, NSM and NFSACL), as well as a separate metric that indicates the number of open NFSv3 connections.

VAST Web UI

ORION-239519: Enhanced the inspection pane in the User Management page (User Management -> Local Users -> select a user and click > to open the right-side pane) to add an indication of whether the S3 access key is local or replicated.

• ORION-193527: Made updates to display the BMC firmware version in the Infrastructure -> EBoxes page. Prior to this change, this information was available in the CNodes/DNodes pages only.

VAST CLI

• Some VAST CLI listing commands, such as viewpolicy list or cluster list-locks, feature a new option, --vertical, that formats the output so that the items and their properties are listed one below another, for example:

    vcli: admin> viewpolicy list --vertical
    +------------------------------------------------+----------------------+
    | ID                                             | 9                    |
    | Name                                           | bgio-mercury:default |
    | Cluster                                        | vast-bnm             |
    | <other properties>                             |                      |
    +------------------------------------------------+----------------------+
    +------------------------------------------------+----------------------+
    | ID                                             | 5                    |
    | Name                                           | my_policy            |
    | Cluster                                        | vast-bnm             |
    | <other properties>                             |                      |
    +------------------------------------------------+----------------------+
  

• Added commands to restart a CNode or a DNode: cnode powercycle and dnode powercycle.

• The cluster list-locks command has a new required parameter that specifies the path for which to list locks: --file-path. It also features a new pagination direction option, --direction.

• The dns create and dns modify commands offer a --port option for you to set the DNS service port.

• The certificate create and certificate modify commands feature new options, --ca-certificate and --cert-type, that you can use to upload a CA certificate and indicate whether the certificate is intended for webhooks or for Kafka connections.

Platform and Control

• Lifted the following limitation on conversion to write buffer RAID:

  • No cluster expansion operation is in progress (for example, DBox expansion, migration, or replacement activities).

• Added support for having both EBoxes and CNodes in the same VAST cluster.

  The Infrastructure -> CNodes page in VAST Web UI has been updated to include a new Position column to help differentiate between a node that is part of an EBox and a regular CNode. CNodes that are part of an EBox are listed as Virtual.

• VAST Cluster 5.4 lifts the limitation on enabling the DBox HA feature on new installations, which was in effect for releases 5.1, 5.2 and 5.3.

Resolved Issues

Install and Upgrade

• ORION-259291: Resolved an issue due to which in some cases, if the Inband option was selected in the Management network field of the VAST Cluster Install wizard, the CNodes could be configured as inband while the DNodes were configured as outband.
• ORION-238278: Made updates to prevent a flow where cluster deployment could fail with the FW parameters have changed. please use ipmitool power cycle to cycle the server and rerun error when running the cluster networking configuration script (configure_network.py) on the cluster nodes.
• ORION-232777: Fine-tuned inter-node SSH connection timeout settings to prevent an OS upgrade from failing due to the ConnectionLost('Login timeout expired') error.
• ORION-229534: Introduced optimizations to shorten the time required to add a large number of nodes to the cluster.
• ORION-225902: Resolved an issue where a DNode was deemed failed during an upgrade because of timeouts that occurred when running internal NVMe CLI commands to update the list of hosts.

Cluster Expansion

• ORION-262506: Resolved an issue that could occur during DBox expansion and cause device activation to take much longer than expected, resulting in expansion failures.
• ORION-253664: Resolved an issue where an attempt to add a MLK DBox hang in the RUNNING state due to an InterfaceError('connection already closed') error.

Networking

• ORION-269072: Enhanced validations of ASN entries made when configuring a BGP connection (Network Access -> BGP Configurations -> Create BGP) to ensure that each value does not exceed 32 bits.
• ORION-258779: Updated the CNode removal task to skip validation of the OpenSM service on clusters with InfiniBand internal networking if forced removal was requested, thus allowing to forcefully remove a failed CNode with no internal network connectivity.
• ORION-252485: Deprecated the Subnet bits and VIP Grace Period fields in BGP configuration settings (Network Access -> BGP Configurations -> choose to create or edit a BGP configuration).
• ORION-241079: Made updates to prevent "port_rcv_switch_relay_errors" increased during the run alerts from appearing in the VMS log.
• ORION-204316: Enhanced the cluster networking configuration script (configure_network.py) to ensure that it properly configures CNodes on HPE IceLake clusters when CNode Port Affinity is used.
• ORION-185008: Resolved an issue where modifying a subnet in an existing virtual IP pool could result in the pool being not accessible by clients.

Element Store

• ORION-268120: Resolved an issue where an unexpected client disconnection during the TCP connection establishment phase could cause the CNode container to restart with the assertion failed: ((*__errno_location ()) == 11 || (*__errno_location ()) == 11) errno: Software caused connection abort error.
• ORION-262912: Made updates to prevent drive latency spikes with subsequent denylisting when processing a specific type of workload.
• ORION-259006: Resolved an issue that could cause CNode containers to restart with the Buffers pool is exhausted error when running replication over an encrypted connection.
• ORION-256617, ORION-221522: Optimized processing of metadata in replication flows to resolve an issue which could cause a CNode container to restart with timeout expired <...> (TRAVIS) or timeout expired <...> (INGEST_WRITE) errors.
• ORION-254716: Resolved an issue that could cause multiple CNode containers to restart with the Address not mapped to object error.
• ORION-250931: Updated VAST Catalog-related optimizations to resolved an issue that could cause multiple CNode containers to restart due to timeout expired <...> (TRAVIS) and spinlock lock takes too long errors.
• ORION-248586: Resolved an issue that could cause occasional failed to insert reference cache key to the references cache generation tree alerts on the cluster.
• ORION-245764: Eliminated a flow that could cause an allocated 90% of mooktze buffers! top consumer is DIRSNAP_RESOLVED_SNAPSHOTS alert on the cluster.
• ORION-245108: Improved handling of existing open handles for deleted files to prevent timeout expired <...> (INGEST_WRITE) errors when processing NFSv4 write workloads.
• ORION-244972: Resolved an issue that could cause a Too many attributes changes protected by snapshots on handle. IO will fail. Need to delete snapshots to continue alert followed by an ESTORE DELETE_SNAP denylist imposed due to the resolver->has_map_for_element(chandle) error.
• ORION-244281: Resolved an issue that could cause multiple CNode containers to restart with the timeout expired <...> (TRAVIS) error.
• ORION-241655: Resolved an issue where after enabling VAST Catalog on the cluster, the ESTORE BIG_CATALOG denylist was imposed following halted write alerts.
• ORION-236508: Eliminated a flow that could cause an allocated 90% of mooktze buffers! top consumer is ESTORE_MIGRATOR_WRITE_HANDLES_TREE alert on the cluster.
• ORION-221633: Eliminated a race condition that could cause a (num_entries_removed == 1) (2 == 1) didn't find name entry to remove when applying content defrag future error followed by the ESTORE CONTEN_DEFRAG denylist on the cluster.
• ORION-214091: Resolved an issue that could cause the (!has_collision) Found two old extents corresponding to same future that overlap each other error followed by ESTORE CONTENT_DEFRAG denylist on the cluster.
• ORION-198889: Updated defragmentation routines to resolve an issue where multiple CNode containers restarted with the shard in release for too long error after adding a highly intensive workload.

Multi-Tenancy

• ORION-252148: Resolved an issue where an attempt to mount an NFSv4.1 view could not succeed on one of the virtual IPs in a virtual IP pool while all other virtual IPs from the same pool worked as expected.

Quality of Service (QoS)

• ORION-231253: Enhanced workload prioritizing to prevent scenarios where the actual performance could be 10% off the cluster-wide write bandwidth limit specified.

SMB

• ORION-249181: Made updates to reduce the time needed to complete NTLM authentication when the Use SMB native authentication option is enabled for the cluster.

S3

• ORION-240463: Optimized a flow that could occur when listing objects in a versioned bucket.
• ORION-188749: Updated the logic behind the view policy option that restricts S3 read/write access based on client IP addresses (in VAST Web UI: Element Store -> View Policies -> choose to create or edit a view policy -> Host-Based Access -> S3 pane -> Read/Write) so that the option supports bucket-level operations, such as creating or deleting a bucket.

Block

• ORION-245989: Made updates so that bulk operations on volumes performed by a cluster admin, can be tracked by a tenant admin in the ​Activities​​ page of VAST Web UI.

Data Protection

• ORION-267226: Fine-tuned Global Namespace timeouts to resolve an issue where creation of a global snapshot clone for a path with a very large number of subdirectories took longer than expected.
• ORION-238060: Eliminated a flow that could cause the cluster to reach the HALT_ALL_INCLUDING_UNLINKS metadata state when cloning a snapshot of a directory with a very large number of files.

Replication

• ORION-267046: Resolved an issue where an attempt to delete a protected path with a replication stream in a CREATE_FAILED state resulted in a DELETE_FAILURE_OBJ_IS_BEING_USED error.
• ORION-256402: Updated handling of sync points in some replication flows to resolve an issue that could cause replication streams to miss their RPOs due to a TOO_MANY_CLONES error.
• ORION-233749: Made updates to ensure automatic deletion of access keys associated with a replication destination tenant which has been deleted.

VAST on Cloud (VoC)

• ORION-230479: Updated the logic behind the Periodic readiness check failed for OS upgrade alarm to avoid raising the alarm on VoC on GCP clusters.

Authentication and Authorization

• ORION-262465: Resolved an issue that could cause a ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS access key Id you provided does not exist in our records - due to mismatch error when trying to access a bucket using S3 keys of a local user that had the same name, UID, group membership and identity policy assignment as a user present in the Active Directory.
• ORION-223833: Enhanced handling of the scenario where the cluster is unable to join an Active Directory domain so that the provider does not get enabled as a result of the discovery process, which could otherwise cause protocol traffic outage.

VMS

• ORION-238083: Updated the logic behind the VMS option to power off an EBox to prevent situations where the power off task is reported as complete while the box is inactive but not powered off.
• ORION-225432: Implemented a filter to avoid offering metrics that are not applicable for the platform when selecting a predefined analytic report in the Analytics -> Predefined Analytics page of VAST Web UI.

VAST Web UI

• ORION-256862: Updated the logic behind the the Password restore delay field in indestructibility settings (Settings -> Indestructibility -> Restore Password) to avoid raising a Please provide datetime in positive [number][unit (y/w/d/h/m/s)] format error on an attempt to modify the field value.
• ORION-255304: Updated the logic behind the Type column in the Infrastructure -> Switches page to allow for filtering by any of the switch vendors displayed in the grid.
• ORION-246972: Updated the logic behind the Add Volumes manually option in the Map Volumes dialog (Data Protection -> Snapshots -> right-click a snapshot and choose Map Snapshot Volumes) so that it starts the volume mapping process as expected.
• ORION-207301: Updated the logic behind the fields that show the unit of measure used for the Minimum object size and Maximum object size filters set fo an existing lifecycle rule (Element Store -> Lifecycle Rules -> choose to view or edit a rule) to resolve an issue that caused these fields to display KB while the original values supplied and used were in bytes.
• ORION-203737: Updated the logic behind the filter in the Box column in the Infrastructure -> CNodes and DNodes pages so that the nodes can be filtered as appropriate.
• ORION-203189: Updated the logic behind the External Netmask field in cluster networking settings (Settings -> Configure Network) to accept IPv6 netmasks.

VAST CLI

• ORION-246115: Made updates to ensure that VAST CLI keyword auto-completion works as expected for hyphenated keywords.

VAST REST API

• ORION-178408: Updated the estimated_read_only_time field returned by the /protectedpaths/ and /protectedpaths/<path ID>/ endpoints of VAST REST API to return a float number instead of a string.

Platform and Control

• ORION-272260: Adjusted the way VMS handles CPU ambient temperature thresholds to avoid raising false alarms on cluster nodes.
• ORION-269167: Made updates to avoid an internal SPDK-related flow that could occur following a SSD failure and cause a CNode container restart on Ceres v2 clusters.
• ORION-259669: Resolved an issue where following an upgrade, the number of available stripes started to decrease, with a stripe is stuck alert was raised.
• ORION-257729: Resolved an issue that could cause multiple CNode containers to restart due to the Retry time exceeded - waiting for a free fiber error on a cluster that had VAST Catalog enabled.
• ORION-250069: Made updates to ensure that replacement of a very large number of SSDs at the same time does not result in CNode containers being restarted due to the after deactivation of drive=<...> dbox=<...> will be at risk to have insufficient drives for mioc state and (load_successful) this might mean that system_format failed earlier <...> or that loading mioc failed due to unavailable dboxes/drives errors.
  ORION-244285: Made updates to avoid occasionally raising the request posted buffers in dev_id=0 for module E are 0 alert following a port shutdown.
• ORION-242111: Provided an additional RDMA keepalive mechanism to prevent failures that could occur when recovering from a DBox HA event due to incorrect CNodes being shut down during the recovery.
• ORION-240717: Resolved an issue where a Failed to decompress LZ4 block: incorrect block header occurred while trying to restore Veeam backups stored on the VAST cluster.
• ORION-236867: Provided graceful error handling to avoid a CNode container restart in case the gateway IP could not be found.
• ORION-236751: Resolved an issue that could cause a CNode container to restart with the ((_memory_client_counter) > (0)) (0 > 0) error when processing NFSv4.1 workload.
• ORION-236567: Made updates to prevent CNode container restart with the Invalid permissions for mapped object - (stack_size=8184 bytes_remaining=0 pct_remaining=0%) HINT: is your stack big enough? error when using S3 with TSL.
• ORION-236431: Improved formatting of error messages related to cluster networking to prevent periodic printing of the NVMe CLI ValueError: not enough values to unpack message in /var/log/messages.
• ORION-236292: Resolved an issue that could cause multiple CNode containers to restart with the timeout expired <...> (SARRAY_READ) error.
• ORION-232944: Updated the logic behind the Power off option in the CNode page of VAST Web UI (Infrastructure -> CNodes -> right-click a node) so that it does not require to be clicked twice to power off the node.
• ORION-231272: Resolved an issue that could cause increased write latency for specific workloads on a Cisco EBox.
• ORION-223130: Resolved an issue where a CNode container restarted with the timeout expired <...> (WRITE_BUFFER_READ) error.
• ORION-197576: Resolved an issue that could cause false Session Audit Bad User PWD alerts in ipmitool sel elist logs.
• ORION-186041: Resolved an issue that could cause a CNode container to restart with the Migrate resulted in two overlapping extents at the same snap + HD!? error.

Callhome and Support

• ORION-238168: Updated the functionality that lets you delete support bundles so that you can delete a bundle that is still in the process of being created.

Limitations

Install and Upgrade

• Automatic drive firmware upgrade is not performed on drives that have been moved manually from an old DBox to a new DBox during the DBox replacement procedure. (Note that this does not apply to EBoxes.)
• ORION-281679: SSD firmware upgrade is not supported for Supermicro EBoxes.
• ORION-280966: Having imported a JSON configuration file in VAST Cluster Install, you need to manually verify that all the populated fields have expected values. Sometimes, depending on the cluster configuration and environment, some of the fields are not populated as expected during the import.
• ORION-242658: BMC firmware upgrades are not supported for Supermicro Genoa CNodes.
• ORION-222648: NDU that includes automatic adjustment of CNode CPU isolation settings (isolcpus) is not supported for EBoxes.
• ORION-214559: A BMC upgrade cannot be performed with an inactive CNode that has been powered off.

Networking

• The following limitations apply when using Open Telemetry-Based Ethernet Network Monitoring:
  • Up to 70 switches per cluster.
  • The switch must have OTel configured to send telemetry data over gRPC to the VMS IP address using port 4317.
  • TLS on the connection used to obtain the metrics from the switch is not supported.
  • Supported on NVIDIA Cumulus switches 5.12.1 and later. For other switch vendors, inquire VAST Support.
• The following limitations apply when implementing L3 networking:
  • After enabling L3 access for a virtual IP pool, it cannot be disabled.
  • L3 access is not supported on virtual IP pools for which CNode Port Affinity is configured.
  • ORION-333743: L3 access is not supported on virtual IP pools that include more than one range of virtual IPs.
  • L3 networking is not supported on IB clusters.
  • L3 networking is not available for VAST on Cloud.
  • MD5 authentication is not supported.
  • ORION-266297: When using numbered BGP, each CNode can only be involved in one BGP configuration. To learn which CNodes are currently used in which BGP configurations, use the Network Access -> Virtual IP Pools page that displays CNodes and BGP Configurations for each virtual IP pool.
  • Reverting an L3-configured cluster to L2 through VMS is not supported. If you need to revert, contact VAST Support.
• Changing network configuration from VMS (in VAST Web UI: Settings -> Configure Network) is not supported for clusters that have any CNodes that are connected simultaneously to multiple separate client data networks.
• ORION-374025: Virtual IP pools with a role of Query Engine cannot be modified after creation. This is a built-in safeguard to prevent in-flight queries from failing. If you need to introduce a change into an existing Query Engine pool configuration, delete the pool and create a new one.

Encryption

• The following rules and limitations apply when using Kerberos providers:
  • MIT Kerberos providers are used for NFSv4 only.
  • A tenant can be associated with a single Kerberos provider only.
  • Up to eight Kerberos providers can be defined on a VAST cluster.
  • Kerberos principals used to authenticate to the VAST cluster must exists as users in an LDAP provider or in a VAST provider associated with the same tenant as the Kerberos provider. Otherwise, they are squashed to the nfsnobody user.
• The default tenant does not use keys created or stored within an EKM. To encrypt data within the default tenant with EKM-based keys, use encrypted paths.
• ORION-388667: Creating an encrypted path on a directory exposed by an ABAC-enabled view is not supported. An attempt to create an encrypted path on the same directory as the one exposed by an ABAC-tagged view causes the assigned ABAC tags to be removed.
• (RESOLVED IN 5.4.4) ORION-341362: The VAST KMIP client does not support the TLS SNI extension.
• ORION-208004: Enabling VAST OS boot drive encryption requires that the node is inactive. Enabling the encryption on an active node may cause a long reboot sequence.

Quotas

• The following limitations apply when using Remote Quota Protocol (rquota):
  • SETQUOTA requests are not supported.
  • Quota information can be provided for active users only.
  • Information about group quotas is not provided.
• Quotas are not enforced on replication destination directories under a protected path. For example, if the protected path is /ppath, a quota on /ppath/yourdir is not enforced.

Quality of Service (QoS)

• Use of QoS with RDMA is not supported.
• The prioritization flag cannot be set for user QoS policies.
• Some high-priority optimizations are applied to NFSv3 only.
• When the cluster-wide maximum write bandwidth is set, the actual performance may be ±15% of the expected performance.
• S3 (including Kafka and VAST Database) and block storage I/Os are not calculated as part of the cluster-wide maximum write bandwidth limit.

NFS

• ORION-115336: If one creates an NFSv4.1-only view and mounts it, and then creates its parent view with NFSv3 only, IO operations on the NFSv4.1-only view succeed but mounts are not allowed.

NFSv4

• TLS encryption with NFSv4.1 is not supported for NFSv.4.1 over RDMA.
• The following requirements and limitations apply when using NFSv4 file delegations:
  • The NFSv4 protocol requires Network Time Protocol (NTP) to be configured on both the VAST cluster and the client. While VAST does not enforce this requirement, the absence of proper time synchronization may, in rare cases, result in unexpected behavior.
  • The client must have an NFSv4 backchannel connection.
  • NFSv4 delegations are not supported with RDMA.
  • For multi-protocol views with NFSv4 delegations enabled, close-to-open consistency is only preserved between NFSv4 clients, but not between NFSv4 clients and other protocols (including NFSv3). For example, if a non-NFSv4 client modifies a file on such a view, the modification may not become immediately visible to an NFSv4 client holding an active delegation on the file.
  • Before making a path read-only by using the /folders/read_only endpoint of VAST REST API, ensure that NFSv4 delegations are disabled or returned. Otherwise, unexpected behavior may be encountered.
  • Restoring a snapshot on a tenant that have NFSv4 delegations enabled, may cause unexpected behavior with regard to delegated files. It is recommended to revoke existing delegations before restoring a snapshot.
  • VAST Cluster recalls granted NFSv4 delegations when a failover process is started.
  • In some cases when there is very high NFSv4 callback load per cluster or per client, VAST Cluster might start recalling or revoking NFSv4 delegations due to callback request timeouts.
  • NFSv4 delegations require mounting the client at least as NFS version 4.1. It is recommended to mount as NFS version 4.2 for additional performance improvements. Linux kernels 6.11+ on a client provide more performance advantages (RFC9754), which get backported to older kernels through VAST-NFS.
  • If you are using VAST-NFS, note that NFSv4 delegations are supported with VAST-NFS 4.0.32 and later. VAST-NFS 4.5 includes enhancements related to support of NFSv4 delegations.
  • SUSE Linux Enterprise Server 12.x and CentOS 7.x clients are not supported with NFSv4 delegations enabled.

SMB

• The following limitations apply when using SMB hardlinks:
  • Creating a hardlink to another SMB-enabled view is not allowed.
  • Creating hardlinks to directories is not allowed.
  • Linking an Alternate Data Stream (ADS) is not supported.
  • Creating a hardlink on a Global Access satellite cluster is not supported.
  • Creating a hardlink, on the Global Access origin cluster, of a file that has already been opened from a satellite cluster, is not allowed.
• Views exposed as SMB shares work only if the cluster is joined to Active Directory. This includes both SMB-only and multiprotocol views.
• The following limitations apply when using SMB change notifications for subdirectories:
  • SMB recursive change notifications are not available for paths for which Global Namespace is configured.
  • Using SMB recursive change notifications for nested SMB shares may entail unexpected behavior.
• The following limitations apply when using Alternate Data Streams (ADS):
  • ADS are not seen or accessible by protocols other than SMB.
  • ADS will not be retained when files and directories are replicated to S3.
    ORION-169707: When the Hyper-V management tool tries to list VAST Hyper-V SMB shares on an SMB server, the The RPC server is not available error can occur if the SMB server is specified using its FQDN. To avoid this error, specify the IP address of the SMB server instead of the FQDN.
• ORION-160323: After updating permissions for an SMB share in Windows Explorer, a duplicate SMB share can be displayed. The duplicate SMB share disappears upon a refresh (F5).
• ORION-134730: An attempt to restore a file can fail if after the restore has started, a quota is set on the path where the file resides.

S3

• The following limitations apply when using S3 Versioning:
  • After versioning has been enabled on a bucket, it cannot be disabled.
  • Multiprotocol access to versioned objects is not supported. NFS and SMB clients may attempt to access the S3 versioned objects in read-only mode.
  • MFA Delete is not supported.
  • S3 conditional writes are not supported for versioned buckets.
  • ORION-288978: S3 versioning is not supported with Global Access.
  • ORION-143808: S3 versioning is not supported with global snapshot clones. An attempt to put a versioned object to a bucket at the global snapshot's destination path fails with an internal error.
• The following rules and limitations apply when using S3 Server-Side Encryption with Customer-Provided Keys (SSE-C):
  • If an S3 request contains an x-amz-server-side-encryption-customer-* header, it is required to use HTTPS for the request to be accepted by the cluster.
  • Only AES256 encryption algorithm is supported.
  • Objects encrypted with SSE-C cannot be accessed through other access protocols.
  • For replication environments, it is strongly recommended to avoid using SSE-C unless all of the clusters involved support SSE-C.
  • ORION-321170: S3 SSE-C is not supported when uploading objects using chunked encoding.
• The following limitations apply when using S3 presigned POST requests:
  • An object to be uploaded via a S3 presigned POST request must have only ASCII characters in its name.
  • A POST policy used for S3 presigned POST requests can be up to 4800 bytes.
• The following limitations apply when using S3 Indestructible Object Mode:
  • An S3 Bucket view with Indestructible Object Mode cannot have other protocols enabled.
  • Indestructible Object Mode cannot be set for a view that points to / (root directory).
  • It is not allowed to have views under the view in Indestructible Object Mode, or at the same path as the Indestructible Object Mode view.
  • Directories that contain views that have indestructible object mode enabled cannot be moved to the trash folder.
  • Indestructible Object Mode cannot be used together with S3 Object Locking or S3 Object Versioning.
  • Indestructible Object Mode cannot be used together with WORM.
  • Indestructible Object Mode cannot be set for a view that exposes the protocol audit log directory.
  • A snapshot cannot be restored on a view that is in Indestructible Object Mode.
  • Views in Indestructible Object Mode are not subject to replication or Global Access.
• ORION-340559: Absolute-form URLs (e.g. http://.../.../<object> instead of /<object>) in CompleteMultipartUpload requests are not supported.
• ORION-240888: S3 bucket monitoring does not take into account VMS-originated S3 requests, including those related to features such as:
  • Lifecycle rules
  • Event notifications
  • Bucket policies
  • Object ownership
  • Bucket versioning
  • Object locking
  • Bucket logging
• ORION-197281: VAST Cluster disables bucket logging set on a bucket from which data is synchronously replicated to another bucket once you set up bucket logging on the replication destination bucket and configure it to use a different logging destination bucket.
• ORION-190674: Once created, an S3 bucket cannot be renamed or moved to a different path. Thus, for example, if you try to change the bucket’s path when modifying a view in VAST Web UI, the change does not take effect and the view will still be listed with the old path.

Block

• VAST Cluster does not support subsystem names that contain a space (in VAST Web UI: Element Store -> Views -> choose to create a view -> Block tab -> Subsystem name field).
• Snapshots on local protected paths are allowed but replication on non-local protected paths is not supported.
• For Rocky Linux-based clients, VAST recommends that the client uses Rocky Linux 9.4 or later.
• An attempt to remove a volume that has snapshots may cause errors for volume objects of snapshots of that volume, if they exist.
• A view that is used to expose block storage cannot have other storage protocols enabled.
• The following VAST capabilities are not available with block views:
  • Access control features (such as ABE, ABAC, WORM)
  • VAST Audit Log
  • Replication to a remote peer
  • Global Access
  • Remote global snapshot clones
• Nesting of a block view inside an existing block view is not allowed.
• The host NQN cannot be modified. To change the NQN, you need to remove the host and then add and map it anew.
• You cannot enable or disable block storage support on an existing view. Block storage support can only be enabled for a view during view creation and cannot be disabled afterwards.
• Block devices can be created on empty directories only.
• If a host defined on the VAST cluster does not have any volumes mapped to it, NVMe auto-discovery does not show this subsystem.
• When using the VAST Web UI or CLI options to bulk create volumes or hosts, the number of items to be created cannot exceed 256. When mapping hosts to volumes, up to 256 items can be mapped at a time.
• The maximum IO block size is limited to 1MB (4GB for unmap).
• (RESOLVED IN 5.4.1) ORION-272773: Automatic validation that enforces uniqueness of host NQNs, does not validate NQNs for the default tenant. Only non-default tenants are checked.

Attribute-Based Access Control (ABAC)

• ABAC is supported on views controlled with SMB, S3 Native and Mixed Last Wins security flavors. ABAC is not supported with NFS flavor.
• ABAC tags cannot be set on the cluster’s root directory (/).
• If a user does not have any ABAC permissions, the user still can mount an NFSv4 export or map a SMB share to a local drive, but the user is not allowed to perform any operations on the files or directories.
• Once assigned, you cannot edit or remove the ABAC tags of a view. Assigning new ABAC tags to an existing view or directory (storage path) is not allowed.
• ABAC is not supported with NFSv3.
• If you create a view for a directory that already exists, ABAC tags from the existing directory are assigned to the newly created view. In this case, there can be a delay between the view creation time and the time when the view's ABAC tags can be displayed.
• After a child view inherits ABAC tags from the parent view, you cannot update or remove the ABAC tags on the child view.
• ORION-388667: Creating an encrypted path on a directory exposed by an ABAC-enabled view is not supported. An attempt to create an encrypted path on the same directory as the one exposed by an ABAC-tagged view causes the assigned ABAC tags to be removed.

Write Once Read Many (WORM)

• You can add additional protocols to a WORM-enabled view, but cannot remove a protocol.
• You cannot configure a WORM-enabled view for NFS/SMB and S3 together.
• Nesting views under a WORM-enabled view is not allowed.
• You cannot disable WORM for the view once it is enabled.
• Bulk permission updates are not allowed on WORM-enabled views.

User Impersonation

• ORION-216379: When VAST protocol auditing is enabled on a user-impersonated view, only UID of the original user is included in the log. The user's login name and SID are not included.

Bulk Permission Updates

• A bulk permission update can run only when the target view (the view exposing the files and directories for which you want to update permissions) is on the same tenant as the template view.
• Only one bulk permission update task per tenant can run at a time.
• Running a bulk permission update on a view where the security flavor does not match that of the template view may result in inaccessible or incompatible permissions set.
• Read-only snapshots and VAST special directories (.vast in S3 buckets, .trash, .snap, .remote) are excluded from bulk permission update.
• If a client attempts to set permissions on directories or files being updated via a bulk permission update, the result is unpredictable.
• Bulk permission update cannot run on ABAC-tagged views.

Event Publishing

• The following limitations apply when using VAST Event Broker:
  • Producer API:
    • Messages are limited to 1MB.
    • In the event record, the key is limited to 126KB and the value is limited to 126KB.
    • Access to topics by UUID is not supported.
    • Idempotent producing is not supported.
    • Automatic creation of topics is not supported.
  • Consumer API:
    • No more than 256 consumer groups per view (broker)
    • The following is not supported:
      • Consumer group stickiness parameters (such as group.instance.id)
      • READ UNCOMMITTED isolation level
      • Cooperative rebalancing
      • Client rack awareness
      • Fetch sessions (only full fetch will be applied), delayed fetch parameters
      • Seek by time
  • Admin API:
    • Supported APIs include the APIs to create topics, delete topics, and to delete groups, as well as describeConfigs and alterConfigs APIs that let you get and update the topic configurations.
  • The following Kafka capabilities are not supported:
    • Over-the-wire compression of messages

    Note: VAST compression of data is supported.

    • Transactions
  • Only one virtual IP pool can be associated with a Kafka-enabled view, providing at least one virtual IP per CNode. Once the view has been created, the virtual IP pool cannot be replaced by another one (but it can be modified if needed).
  • The amount of VAST Event Broker views that you can create on a VAST cluster, is limited by the maximum number of views supported by the cluster and by the maximum number of virtual IP pools (since each broker view requires a dedicated virtual IP pool). See VAST Cluster Scale Guidelines for details.
  • The amount of VAST Event Broker views that you can create on a VAST cluster, is limited by the maximum number of views supported by the cluster.
  • The amount of event topics that you can create on a VAST cluster, is limited by the maximum number of tables per VAST Database table and the overall amount of event topic partitions.
  • A topic can have up to 20,000 partitions. The number of partitions in a topic cannot be changed after the topic has been created. Up to 200,000 partitions are supported per VAST Event Broker view.
  • Event queries based on the topic partition are not supported.
  • When listing consumer groups, the response is limited to 256 groups per Kafka-enabled view.
  • Event publishing and consuming operations, as well as topic management operations are not subject to VAST Protocol Auditing or Quality of Service (QoS).
  • VAST Cluster supports Confluent Kafka Python client 2.4 - 2.8. The aiokafka client is not supported.
  • Authentication and authorization:
    • Active Directory/LDAP is not supported. The user must be defined as a VAST local user.
    • Only one Kafka TLS certificate can be uploaded per VAST cluster.
  • Data protection:
    • The Kafka-enabled view needs to be manually created and associated with a virtual IP pool at the destination peer. The pool must have the same name as the one at the source peer.
    • Fast restore of a protected path containing a Kafka-enabled view is not allowed.
    • VAST replication of consumer groups is not supported. Consumer group offsets are not replicated.

VAST DataBase

• The following limitation applies to vector search:

  • VAST Cluster 5.4 does not include performance optimization for vector operations.

• The VAST connector for Trino does not support the SECURITY clause in CREATE VIEW statements.

• The following limitations apply when using table views:

  • View properties are not supported.
  • Queries to a view must include full table names.
  • Redefining a view is supported for Spark clients only.
  • User-defined column names and comments are lost if the schema of the query changes when redefining a view.

• The following limitations apply to sorted tables:

  • Once enabled for a table, the sorting cannot be disabled.
  • Sorting can be performed for tables that contain 512k or more rows. Although you can enable sorting on a table regardless of its size, smaller tables do not get sorted.
  • Once sorted, tables cannot be unsorted or have their sorted columns changed.
  • Only one sorting order can be defined on a table. Up to four sorted columns are supported.
  • Sorted column values cannot be updated to different values.
  • Sorted tables cannot be replicated. Tables that are replicated cannot be sorted.
  • Sorting cannot be enabled on tables that have semi-sorted projections. Semi-sorted projections cannot be added to sorted tables.
  • Nested data types are not supported for sorted columns.
  • Tables that expose the internal row ID with the vastdb_rowid column cannot be sorted.
  • When calculating the sorting status, a constant value within the Sorted value range is returned for tables that contain less than 5 million rows.

• The following requirements and limitations apply when using row/column-level security:

  • Row filtering and column masking are supported for Trino query engines only.

  • The Trino user's identity policy must have the EndUserImpersonation and GetRowColumnSecurity actions allowed.

• A VAST Database view (a view with the Database protocol) can only be created on clusters with data reduction enabled.

VAST Catalog

• The maximum path length supported by VAST Catalog is 1024 characters.
• ORION-197741: VAST Catalog cannot be enabled on a cluster that uses encryption keys managed through EKM, including per-tenant and per-path encryption keys.

VAST DataEngine

• The following rules and limitations apply when running Trino clusters on VAST:
  • Not less than three CNodes are required per Trino cluster (one for the coordinator, two or more for the workers).
  • In case of CNode HA events, the Trino cluster remains accessible and running queries as long as one coordinator CNode and one of the worker CNodes are up.
  • A VAST cluster upgrade (NDU) performed while running a Trino cluster may cause Trino query failures. The Trino cluster will be fully operational after the NDU is complete.
• If VAST DataEngine is enabled on the cluster, replication of the tenant's root directory is not allowed.
• Up to 32 engines are supported per VAST cluster.
• (​​RESOLVED IN 5.4.1-SP5​​) ORION-302749: When using an external Kubernetes cluster, it is recommended to implement the following node-level configuration to prevent the pods from occasionally failing due to the ​Too many open files​​ error:
  ​ 1. Configure ​containerd​​/​docker​​ on all worker nodes to increase the default file descriptor limit:
  ​
  # /etc/systemd/system/containerd.service.d/override.conf
  [Service]
  LimitNOFILE=1048576:1048576
  ​​
  2. Run ​systemctl daemon-reload && systemctl restart containerd​​.

Data Protection

• When using NFSv3, in rare cases with large numbers of files and directories, the existence of a view with Global Synchronization enabled under a protected path can block the removal of the protected path.

Replication

• The following limitation applies to VAST Database asynchronous replication:
  • ORION-179909: VAST Database asynchronous replication cannot be used together with Global Access or synchronous replication on the same path.
  • Only committed database transactions are replicated.
  • VAST Catalog and audit log are not replicated.
  • Multi-tenant replication of VAST Databases is not supported.
• The following limitations apply to synchronous replication:
  • A protected path using synchronous replication can only contain S3 buckets. You cannot use synchronous replication for paths that are exposed to any other protocols.
• The following limitations apply to synchronous replication for S3:
  • Synchronous replication is supported for S3 buckets only.
  • It is not allowed to configure local snapshots, global snapshot clones or Global Access on the protected path for which synchronous replication is configured.
  • S3 lifecycle rules are not replicated.
  • S3 keys are replicated asynchronously.
  • Synchronously replicated directories are not subject to bulk permission updates.
• Protected paths with asynchronous replication cannot be nested.
• Data cannot be moved into or out of a path that is protected by either asynchronous replication or S3 replication.
• it is not possible to change which protection policy controls any replication stream.
• ORION-208123: Local user accounts are not subject to replication.

Global Access

• VAST Database and VAST Event Broker (Kafka) views are not supported.
• The following limitations apply when using Global Access for S3 buckets:
  • Identity policies must be enabled at the cluster to which they get replicated.
  • The following VAST capabilities are not supported on destination buckets:
    • S3 event notifications
    • S3 Indestructible Object Mode
    • Lifecycle policies
    • Write Once Ready Many (WORM)
  • Bucket logging is only supported if both the source and destination buckets are in the same protected path.
  • Bucket replication between two clusters is only supported when the bucket is associated with the default S3 view policy.
  • S3 endpoints are not replicated.
• The following combinations of access protocols and security flavors are supported:
  • Access to a Global Access protected path (global folder) on the source cluster: NFSv3, SMB, S3 with any of the flavors; NFSv4 with any flavor except the SMB flavor.
  • Access to a satellite path on the destination cluster: NFSv3 with NFS or S3 Native flavor, SMB (all flavors), S3 (all flavors)
  • Access to a path which is part of both Global Access and asynchronous replication setup: NFSv3 with NFS flavor, SMB (all flavors), S3 (all flavors)
    If a view is configured with both NFSv4 and SMB, it must be controlled with the NFS security flavor.
• Lease expiration time can only be set when creating a global access protected path. You cannot change lease expiration time when you modify a global access path.
• VAST Catalog does not provide information on the cached data on the remote cluster.
• ORION-194805: Applications that use SMB2 Byte Range Locks are not supported when the SMB client is connected via a remote Global Access protected path. Examples of such applications are Microsoft Office suite on macOS, Microsoft Hyper-V, AutoDesk 3ds Max and some Adobe Premiere plugins.
• ORION-194613: If some files have additional hardlinks, the amount of bytes reported as prefetched can be higher than the actual amount prefetched.

VAST DataSpace

• VAST DataSpace requires that each cluster participating in the inter-connection is running VAST Cluster 5.0 or later.
• VAST clusters with IPv6 management IPs cannot be added to VAST DataSpace. Only IPv4 is supported.
• ORION-135966: The inter-connecting clusters must have connectivity to each other through the clusters' management networks.

VAST on Cloud (VoC)

• VAST Cloud (Polaris) UI does not provide an option to run a cluster upgrade. Cluster upgrades are to be performed using the VAST Web UI.
• VAST on Cloud clusters do not support OS upgrade.
• In the event of downtime, data is rebuilt while the cluster comes back online. In case of a subsequent failure during the rebuild, data integrity is not guaranteed.
• ORION-327048: For VoC on GCP, it is recommended to have a multiple of 8 non-VMS VMs per cluster, since GCP has 8 failure domains and imbalance may result in failed stripe allocations.
• ORION-145141: Creating a tenant with EKM encryption is not supported on VoC clusters.
• ORION-113036: After you reregister the same VoC cluster in Uplink, information about the previously registered instance of this cluster is no longer available in Uplink.

Authentication and Authorization

• The following rules and limitations apply when using STS, IAM roles and OIDC providers:

  • STS is supported with HTTPS only.
  • Use of STS requires that the S3 protocol is enabled for the cluster.
  • One OIDC provider per VAST cluster tenant.
  • When replicating IAM roles:
    • ORION-278747: IAM role replication may take several minutes to complete in case the cluster is under high load or there is a very large amount of roles to replicate.
    • STS access keys are replicated only when the protection path is in SYNC state.
    • ORION-278217: When processing the assume role operation, the cluster may be raising a Failed to replicate STS temp keys alert until the synchronous replication stream/protection path reaches the SYNC state.
  • ORION-287563: Only five OIDC keys can be returned for a manual request to get or refresh OIDC keys.

• The following limitations apply when using netgroups:

  • Hosts should have both forward and reverse DNS entries. When VAST Cluster gets the netgroup hostname response from a NIS or LDAP server, it resolves the hostname via DNS.
  • Netgroups are only used to allow or deny clients' access via NFS. VAST Cluster does not accept netgroup entries in host-based access rules for other access protocols.

• The following limitations apply to Multi-Forest Authentication:

  • VAST Cluster does not allow adding two different Active Directory configuration records with the same domain name but different settings for multi-forest authentication and/or auto-discovery.
  • Names of users' domains are not displayed in data flow analytics.
  • If a trusted domain becomes unavailable and then recovers, SMB clients can use it to connect to the VAST cluster only after a period of time, but not immediately upon domain recovery.
  • Clients cannot establish SMB sessions immediately after a trusted domain recovers from a domain failure.
  • If a group exists on an Active Directory domain in a trusted forest and the group scope is defined as DomainLocal, VAST Cluster does not retrieve such a group when querying Active Directory, so members of such a group are denied access despite any share-level ACLs that can rule otherwise.
  • If TLS is enabled, the SSL certificate has to be a CA-signed certificate that is valid for all of the domain controllers in all trusted forests. If the certificate is not valid for a domain controller, this domain controller is not recognized.
  • ORION-156168: In a multi-forest environment, after migrating a group account from the forest of the cluster’s joined domain to another forest, information about historical group membership is not kept, so users in the migrated group might not be able to access resources to which they used to have access prior to the migration.

• The following limitations apply when using Kerberos/NTLM authentication:

  • ORION-143944: When using Kerberos/NTLM Authentication to authorize SMB users from non-trusting domains, the DOMAIN\username format cannot be used to specify users of remote domains. The username@domain format must be used instead.
  • ORION-141763: Before enabling or disabling NTLM authentication, you need to leave the cluster's joined Active Directory domain. After NTLM authentication is enabled or disabled, rejoin the domain.
  • ORION-134299: When the tenant is set to use Kerberos/NTLM authentication to authorize SMB users from non-trusting domains, both NFS and SMB must use the native SMB authentication (Kerberos), and not Unix-style UID/GIDs.

• ORION-285727: In some rare cases when IAM roles are subject to replication, if you delete an IAM role on a replication source, the deleted role gets recreated as a result of the replication. The recreation might take place, for example, in case a HA event occurs during role deletion, or if you modify the role on the replication destination (at the same time as the role is being deleted on the replication source). If you encounter this issue, delete the recreated role again both on the replication source and destination.

• ORION-202335: If the cluster has Active Directory domain auto-discovery enabled, the discovered domains are kept in cache for quite a long time. If you modify an existing provider's configuration while auto-discovery is on, VMS may still report the old cached entries. To avoid this, rerun auto-discovery or remove and re-add the provider.

• ORION-195524: Following a cluster recovery and while the Active Directory provider is still inaccessible, VAST Cluster can resume IO of provider users if they use NFSv3 or NFSv.4.1 with NTLM authentication. IO of provider users accessing through SMB or NFSv4.1 with Kerberos authentication is not resumed during this period.

• ORION-187136: Identity policies are replicated as disabled to the destination peer, where if needed, they can be enabled manually.

• ORION-152475: An access denied error is returned for NFSv3 or NFSv4 requests if they are checked against an identity or bucket policy with an s3:ExistingObjectTag condition statement in it.

VMS

• Tenant client metrics can only be collected for NFSv3 and NFSv4.
• The following limitations apply when installing a SSL certificate for VMS:
  • Only RSA-generated public keys are supported.
  • Password-protected private keys are not supported.
• When setting a VMS login banner text via the VAST CLI, multiple lines are not supported.
• ORION-212118: If a wrong VMS token is passed, the cluster responds with 403 FORBIDDEN but not with 401 UNAUTHORIZED.
• ORION-187584: An empty realm (which does not contain any objects) cannot be assigned to a role.
• ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.

VAST Web UI

• The visual identity policy builder in VAST Web UI (User Management -> Identity Policies -> choose to create or edit an identity policy) does not include predefined action statements for Kafka-related objects.
• ORION-249325: When you create the first block view on the cluster via VAST CLI or VAST REST API, the VAST Web UI does not display the Element Store -> Volumes or Hosts tabs until you refresh the page.

VAST REST API

• In VAST Cluster 5.4, the response to a GET request sent to the /topics/ endpoint contains only the database name and topic name fields. It does not include other fields that were available in VAST Cluster 5.3.

Platform and Control

• The following limitations apply to conversion to write buffer RAID:

  • Conversion from VAST releases prior to 3.4 is not supported.

  • This capability is not supported for clusters with TLC drives, and also for VAST on Cloud clusters.

  • The cluster must include the following minimum number of DBoxes:

    DBox Type	DBox HA enabled	DBox HA disabled
    Ceres	15	4
    Mavericks	22	4

• The following rules and limitations apply when using Rack-Level Resiliency:
  • Every DBox must be associated with a single failure domain.
  • At least seven failure domains must be defined.
  • The number of DBoxes in the rack cannot exceed the number in any other rack by more than one.
  • The total available DBox capacity in a rack cannot be more than twice the available capacity in any other rack.
  • If two of the racks in a rack level-resilient cluster go down, bringing up only one of them does not return the cluster to normal operation. The cluster restores service only after the second of the racks goes up.
• Use of flash write buffers with DBox HA capability enabled requires at least 10 boxes.
• The following limitations apply to EBoxes:
  • ORION-193794: Power cycling of an EBox where the leader was running may result in significant IOPS degradation until the EBox is up again. Contact VAST Support for a workaround.
  • DBox migration is not available for EBoxes.

Callhome and Support

• An attempt to delete a support bundle before it is completely created may cause unexpected behavior.
• (RESOLVED IN 5.5.0) ORION-255750: The <bundle>.tar.progress file is not removed automatically when the bundle upload is finished.

Known Issues

Install and Upgrade

• (RESOLVED IN 5.4.1-SP5) ORION-329066: An assertion failed: ((bytes_consumed_by_deserialize) == (serialized_size)) error may occur on an attempt to activate a CNode on a cluster that used to be configured with identity policies in version 5.0 and is now being upgraded to 5.4.
• (RESOLVED IN 5.4.4) ORION-299826: When resuming an upgrade of an HPE IceLake CBox with the BMC upgrade flag not set, the upgrade is initiated on all the CNodes, including those that are already at the target version.
• ORION-270710: For some NVIDIA Mellanox NICs, the firmware upgrade process may take much longer than expected. Sometimes, a VMS timeout alert may be raised but the process still completes successfully. If a flint upgrade failure occurs during the process, try power-cycling the node.

Cluster Expansion

• (RESOLVED IN 5.4.4) ORION-321148: Duplicate CNode entries may be created in the VMS in case the VMS gets restarted during a CNode add procedure.
• (​​RESOLVED IN 5.4.3​​) ORION-276386: When adding a Ceres v2 DBox to the cluster using the ​dbox add​ command of VAST CLI, the VMS raises a ​HW_VALIDATION - dbox-<...> - None: invalid "dbox min num of hosts". (observed=2, expected: eq 4)​​ alert although the operation completes successfully.
• ORION-220738: In some cases, VMS does not provide any alerts or other status indication when a drive gets disabled while the newly added DBox is being initialized.
• ORION-175762: In some cases, a DBox expansion procedure run on a cluster with similarity-based data reduction enabled can take longer than expected.

Replacements

• (RESOLVED IN 5.4.6) ORION-369387: After replacement of the CNode in a single-node CBox, the VMS can continue to report the CBox serial number of the previous CNode.
• (RESOLVED IN 5.4.4) ORION-341642: Attempts to run DBox replacement or migration procedures on clusters with both APEX and MLK DNodes fail with the Operation not permitted because following DBoxes are in APEX/MLK mixed state error.

Networking

• (RESOLVED IN 5.4.4) ORION-327378: Specifying an incorrect virtual IP configuration when creating or modifying a virtual IP pool may cause a CNode container restart.

Element Store

• (RESOLVED IN 5.4.1-SP5) ORION-337258: Applying a view policy where the host access rules include hosts' DNS names may cause multiple CNode containers to restart with the assertion failed: ((__left) != (__right)) (0 != nullptr) error in case the DNS names cannot be resolved, for example, due to a DNS server failure. If you encounter this issue, contact VAST Support for a workaround.
• (RESOLVED IN 5.4.4) ORION-249138: An attempt to delete a child directory under an active protected path may cause an ESTORE TREE_UNLINKER denylist alert.

Multi-Tenancy

• ORION-279664: The names of customized analytic reports created by a tenant admin, can be seen by other tenant admins. This issue affects only the report names; the data within the reports are not accessible from other tenants.

Quotas

• ORION-387587: A 503 Forbidden error is returned when a tenant admin tries to create or edit user quotas on directories within the tenant, regardless of the permissions configured for the tenant admin manager user in the VMS.
• (RESOLVED IN 5.4.6) ORION-344126: The /quotas/ endpoint returns raw values for used effective capacity, which may in some cases become negative.

Quality of Service (QoS)

• (​​RESOLVED IN 5.4.3​​) ORION-281100: In some cases, a non-zero QoS wait time can be reported for a user while there are no QoS policies in effect for that user. The issue has no impact on the actual IO by the user.
• ORION-236122: Intense read workloads may impact performance on views controlled with a QoS policy with the prioritization flag set.
• ORION-139913: When applying a QoS policy to NFSv3 access, both data and metadata are taken into account in QoS limit calculations, while with NFSv4.1, only data are considered.
• ORION-137986: Enabling a QoS policy for a view on which a mixed (read and write) workload runs, can result in decreased performance for the workload.

NFS

• (RESOLVED IN 5.4.6) ORION-354911: The output of the showmount command issued against a VAST NFS export is truncated to return 128 host entries only, although the view policy may include more than 128 hosts configured.
• (RESOLVED IN 5.4.4) ORION-318379: When creating subdirectories through NFS, some of the subdirectories may sporadically fail to inherit the default ACL from the parent directory.
• (RESOLVED IN 5.4.0-SP2) ORION-295189: When an NFS client running Rocky 8 or later, creates a child directory in a parent that has default ACLs of rwx for user, group and other, the child directory gets incorrect permissions of other::r-x instead of other::rwx.

NFSv4

• (RESOLVED IN 5.5.0) ORION-238708: In some cases, an NFSv4.1 client attempting to move files to the trash folder may get a permission denied error due to an issue that may cause the trash folder to use a more restricting policy than expected.

SMB

• (RESOLVED IN 5.4.1-SP5) ORION-323152: macOS clients may occasionally encounter an RPC struct is bad error when moving, deleting or renaming files via SMB, although the operation completes successfully.
• ORION-142968: If a quota is exceeded during the process of coping a file to the VAST cluster, the copying process is stopped with a misleading error message: A device attached to the system is not functioning.

S3

• (RESOLVED IN 5.4.4) ORION-345959: The VAST cluster sends message-body data in 304 Not Modified responses, which may cause issues on the client.

• (RESOLVED IN 5.4.1-SP5) ORION-338771: Processing S3 presigned POST requests with specific contents may occasionally cause a CNode container restart. If you encounter this issue, contact VAST Support for a workaround.

• (RESOLVED IN 5.4.4) ORION-329738: Unexpected Content-MD5 header for request type alerts can be raised in case the VAST cluster receives a Content-MD5 header for unexpected request types (e.g. GET). If there are no other symptoms, these alerts can be ignored.

• (RESOLVED IN 5.4.1-SP4) ORION-329277: Running a VAST cluster with S3 bucket logging enabled may cause existing S3 connections to get stuck and new connections to get rejected due to the cluster's S3 connection limit reached.

• (RESOLVED IN 5.4.1-SP5) ORION-324233: The VAST cluster can encounter a flow where CompleteMultipartUpload requests can get stuck, causing the CNodes to reject new S3 connections with an IO is stuck - should close connection alert raised.

• (RESOLVED IN 5.4.1-SP4) ORION-284685: In high-load network environments, S3 PutObject requests with larger TLS record sizes (such as those initiated with Boto3 1.40.42 and later) may require several attempts to be processed or encounter connection timeouts.

Block

• (RESOLVED IN 5.4.1-SP4) ORION-315889: In some cases, an attempt to perform bulk mapping of block hosts to volumes may return Error 409 Client Error: Conflict for url: <...> Try to create block object that already exist while the VMS is not reporting any existing mappings for the affected volumes.

Attribute-Based Access Control (ABAC)

• ORION-196170: When a parent and child NFSv4.1 view both have same ABAC tags on them, an attempt to mount the child view may result in a permission denied error. If this occurs, try setting the ABAC tags for the machine account that the client uses to mount the view.

Event Publishing

• (RESOLVED IN 5.4.4) ORION-355264: Following an upgrade to version 5.4 of a cluster with a VAST event broker configured, the VMS can raise alerts indicating that the broker cannot be found or is not configured on the cluster. The issue is caused by a change in broker naming logic. If you encounter this issue, contact VAST Support for a workaround.
• (RESOLVED IN 5.4.4) ORION-336440: When deleting an event topic, VAST Event Broker does not delete the consumer group offsets associated with the deleted topic.
• (RESOLVED IN 5.4.4) ORION-330261: In case consumer group members do not perform a graceful disconnect (e.g. do not explicitly leave the group), their member entries are not removed from the VAST cluster, preventing the cluster from providing timely response when other consumers are attempting to join the group.
• (RESOLVED IN 5.4.1) ORION-293900: When streaming S3 event notifications or VAST DataEngine function-triggering events to a VAST Event Broker view on a cluster with multiple VAST Event Broker views belonging to different tenants, some of the event notifications may encounter a flow that prevents them from being sent to their destination. If this occurs, an Internal Kafka target supported only a single tenant GUID, got 2 alert will be raised. With VAST DataEngine enabled, this issue may result in some functions not being triggered as expected.

VAST DataBase

• (RESOLVED IN 5.4.4) ORION-328362: Using improper column types in a VAST Database Row and Column Security configuration may cause the CNode container to restart with the ​must call is_fixed_tabular_data_type() for tabular data types only​​ error.
• (RESOLVED IN 5.5.0) ORION-278671: The size displayed for a projection table in the VAST Database UI (DataBase -> *VAST Database -> navigate to the table) may deviate from the capacity used by the table according to the Element Store -> Views page. A discrepancy of up to 10% can be observed.

VAST DataEngine

• (RESOLVED IN 5.5.0) ORION-333926: If the cluster admin generates keys for a local user during the time when the user is logged in to the VAST DataEngine UI (using the old keys), the newly generated keys become valid only after the VMS key cache entry expires. This issue does not occur when the new keys are generated by the users themselves.
• (RESOLVED IN 5.5.0) ORION-292066: In VAST DataEngine UI, when trying to connect to a Kubernetes cluster from a VAST tenant that has upper-case letters in its name, the request fails with an Invalid value: <...>: a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character error.
• (RESOLVED IN 5.4.1) ORION-288909: When trying to list logs or traces in VAST DataEngine CLI, the --tenant option on the CLI command does not work as expected.

Data Protection

• (RESOLVED IN 5.4.4) ORION-348223: Clients accessing a cluster which is part of Global Namespace, may encounter intermittent remote IO errors on NFS read operations unless the view policy is set to never update the atime.
• (RESOLVED IN 5.4.6) ORION-342541: When a protection policy has the time to keep local copies set to 0 (zero), the cluster may raise false alarms on missing the PRO target.
• (RESOLVED IN 5.4.4) ORION-323129: A validation that prohibits creation of a protected path on a directory that is used as a destination for a global snapshot clone, inadvertently applies the restriction to same-named directory paths under other tenants (not involved in the global snapshot clone setup).

Replication

• ORION-393875: An attempt to add a third member to a replication group may got stuck if the two destination peers in the group run different VAST releases, as follows:
  • One peer is on version 5.4 or later, while the other is on a version earlier than 5.4.
  • One peer is on version 5.1 or later, while the other is on a version earlier than 5.1.
• ORION-391617: Simultaneously creating S3 buckets on 2 clusters configured to run synchronous replication with Bucket Replication enabled, where the bucket paths are being synchronously replicated to the other cluster, may fail. Either create both buckets on one cluster, or wait until the first bucket creation completes before starting the second.
• ORION-140894: When attempting to delete a protected path from the destination peer after an ungraceful failover, a Failed to delete following streams or a similar error occurs. The workaround is to manually change the destination peer's role to STANDALONE and retry the deletion.

Global Access

• ORION-145307: Bulk permission updates are not supported for files and directories on satellite clusters.

Authentication and Authorization

• (RESOLVED IN 5.4.4) ORION-339595: Rotation of Active Directory machine account credentials may in some cases cause subsequent failures in Kerberos authentication of SMB access. If you encounter this issue, unjoin and rejoin the Active Directory domain.
• (RESOLVED IN 5.4.4) ORION-335361: Domain discovery failures may occur in a two-forest environment where the trust relationship is established between child domains (one from each forest) but not between the forests' root domains.
• ORION-25479: Latin characters are not supported with LDAP. If you attempt to pass, for example, a username encoded with a Latin character set, the LDAP sanity check res: Invalid credentials error is returned.

VMS

• (RESOLVED IN 5.4.6) ORION-375435: The /api/folders/ endpoints return error code 503 Service Unavailable instead of 404 Not Found in case the folder, its path, user or group could not be found when creating a folder, modifying a folder, deleting a folder, or obtaining the folder statistics.
• (RESOLVED IN 5.4.6) ORION-366846: If the cluster's VMS has been upgraded to version 5.4 but the CNodes are still running version 5.3, quota metrics are not available. If you encounter this issue, contact VAST Support for a workaround.
• (RESOLVED IN 5.4.6) ORION-363371: When using the Save & Test button in the event definition dialog of VAST Web UI (Alarms and Events -> Event Definitions -> choose to edit an event definition), the webhook data variable ${vastdata_cluster_name} is not substituted with a real cluster name.
• (RESOLVED IN 5.4.6) ORION-355322: In some cases when the VMS cannot successfully complete the hardware discovery process and defaults to using Supermicro Gen5 as the box type, the VMS can get stuck without being able to restart. If you encounter this issue, contact VAST Support for a workaround.
• (RESOLVED IN 5.4.6) ORION-333283: No email notifications are sent for events for which the Alarm only option is enabled, although the corresponding alerts are present in the VMS and email notifications are configured as appropriate. If you encounter this issue, contact VAST Support for a workaround.
  ORION-332874: Cluster's attempts to send email notifications through SMTP may fail with a certificate verify failed: self-signed certificate (_ssl.c:1006) error. If you encounter this issue, contact VAST Support for a workaround.
• ORION-329549: In some cases after a cluster upgrade or expansion, the Analytics -> Data Flow and Analytics -> Top Actors pages of VAST Web UI may show the No data to display error instead of actual workload data. If you encounter this issue, contact VAST Support for a workaround.
• (RESOLVED IN 5.4.6) ORION-321410: VMS shows the position of both PSUs on Supermicro Turin nodes as Right.
• (RESOLVED IN 5.4.4) ORION-316875: If a virtual IP pool has been renamed, the pool-related metrics obtained through VAST Prometheus Exporter may still show the old pool name.
• ORION-300508: Some metrics may not be available in VAST Prometheus Exporter. Missing metrics include (but may be not limited to) CNode and DNode hardware metrics or CPU temperature, utilization, memory percentage, PCI error count, retransmitted segments, total correctable/uncorrectable memory errors. If you encounter this issue, contact VAST Support for a workaround.
• (RESOLVED IN 5.5.0) ORION-293054: VAST Prometheus Exporter exports duplicate entries of vast_fan_metrics_hardware_rpm and vast_fan_active metrics for Ceres v2 DNode fans.
• ORION-292325: When creating a customized analytics report for block volumes or hosts, some metrics intended to reflect a maximum latency are set to a fixed value when there are no block volumes on the cluster.
• (RESOLVED IN 5.4.1) ORION-291686: The read and write latency shown in the Analytics -> Data Flow page of VAST Web UI is calculated without taking into account the amount of I/O on each of the cluster nodes.
• (RESOLVED IN 5.5.0) ORION-282603: User-made changes to the csi role (one of default administrative roles on the cluster) are not preserved during an upgrade.
• ORION-143717: On a cluster with CNode Port Affinity configured, there is no way to expose the VAST DNS IP on a specific port (left or right).
• ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.
• ORION-89570: In some cases, capacity analytics for subdirectories cannot be reported due to an internal timeout. This issue occurs when there is an extremely large number of subdirectories to be estimated.

VAST Web UI

• (RESOLVED IN 5.5.0) ORION-355178: The ​Acknowledge Filtered​ button in the ​Alarms and Events​​ page does not acknowledge the selected alarms.
• ORION-308654: When using the Export selected rows as CSV option of the VAST Web UI, the resulting file may contain all of the rows, including those that were not selected for the export.
• ORION-294888: The Infrastructure pages of VAST Web UI may display [object Object] instead of the actual CPLD/UBM values for Dell Turin CNodes.
• (RESOLVED IN 5.4.1) ORION-290976: When creating an Active Directory or LDAP provider configuration on the VAST cluster, the attribute mapping fields are read-only. To edit these fields, save the provider configuration, then open it for editing and update the fields as needed.
• (RESOLVED IN 5.4.1) ORION-281960: The Activate Write Buffer RAID button (Settings -> Cluster -> General Cluster Setup and Actions -> Write Buffer RAID pane) is not grayed out when the feature has already been activated on the cluster.
• (RESOLVED IN 5.5.0) ORION-260502: The Capacity Estimation page (Analytics -> Capacity), which is designed to display capacity for directories or, in case of a VAST Database, for its schema, sometimes also includes VAST Database tables in capacity estimations.
• ORION-239505: The VAST Web UI toggles for selecting operations to be logged in the syslog (Settings -> Notifications -> Syslog Setup) do not enforce logging of the selected operation types as expected. If you encounter this issue, contact VAST Support for a workaround.
• ORION-234835: Some VAST Web UI pages might not allow for proper filtering or sorting by column where value presentation differs from that in the VAST internal database.
• ORION-203189: The External Netmask field in cluster networking settings (Settings -> Configure Network) does not accept alphabetic characters.
• (RESOLVED IN 5.5.0) ORION-175189: When querying a local user using the Aggregated context, the Leading GID and Primary group SID fields in the User Details dialog have a value of -1 instead of an empty string.

VAST CLI

• (RESOLVED IN 5.4.6) ORION-332126: The lifecyclerule modify command accepts the following options only when they contain an underscore in the middle of the keyword: --min_size, --max_size, --expiration_days. If the option name contains a hyphen instead of an underscore, the unrecognized argument error occurs.
• ORION-322120: When displaying path names in VAST CLI, the names written in right-to-left languages may appear spelled from left to right.
• (RESOLVED IN 5.4.4) ORION-282853: In some cases, changing the view path by using the view modify --path <new path> command does not work as expected. The command may seem to succeed initially but automatically revert back to the original path within minutes.
• (RESOLVED IN 5.5.0) ORION-269350: An illegal argument error occurs when trying to run the protectedpath list command with the --protection-policy-name option specified.
• ORION-265720: VAST CLI auto-completion does not include the --detach-krb-provider option on the tenant modify command.
• (RESOLVED IN 5.5.0) ORION-156628: An attempt to run the viewpolicy show --audit command results in a 'ViewPolicyProtocolsAudit' object has no attribute 'get' error.

VAST REST API

• ORION-178569: The /users/names endpoint always returns only the first 50 entries, regardless of the page size parameter or the total amount of entries to be returned.

Platform and Control

• (​​RESOLVED IN 5.4.3-SP2​​) ORION-355956: In rare cases, the cluster may encounter a high amount of DNode memory ECC errors, causing the leader process to enter repeated failure cycles and resulting in cluster instability.
• (RESOLVED IN 5.4.3-SP1) ORION-353850: A HA event on the cluster's leading node may in some cases result in inability to resume IO after the event due to the new leader not redistributing virtual IPs to the CNodes. This issue can only be encountered with CNodes that are active do not have northbound connectivity (e.g. dual-NIC CNodes where the northdown NIC is down).
• (RESOLVED IN 5.4.4) ORION-344108: Following an upgrade to VAST Cluster 5.4, a DNode HA event may result in multiple container restarts and subsequent service interruption. To avoid encountering this issue, contact VAST Support.
• (RESOLVED IN 5.4.4) ORION-343501: IO errors occurring on SSDs may cause deactivation of SCMs on the same DNode (and vice versa, errors on SCMs may cause SSD deactivation), resulting in increased IO latency and unnecessary temporary disconnection for the users.
• (RESOLVED IN 5.4.4) ORION-327357: For Dell Turin CNodes, the VMS may show the status of the PSUs as ​UNKNOWN​​.
• ORION-275610: The Drive Compatibility page in VAST Web UI (Support -> Drive Compatibility) does not display all of the details that are available when running a supporteddrives list or supporteddrives get command of VAST CLI.
• ORION-255054: Once a DNode replacement operation is complete, the old node may still be listed in the Infrastructure -> DNodes page of VAST Web UI for some minutes.

Callhome and Support

• (RESOLVED IN 5.4.6) ORION-269349: When obfuscating a support bundle, the obfuscation is not applied to files that have already been zipped.
• ORION-239170: When obfuscating a support bundle, the CNode hostname may not get obfuscated in some of the logs included in the bundle.