VAST Cluster 5.5.0 Release Notes

To access this release, contact the VAST Customer Success team via a support ticket, Slack, or by sending an email to support@vastdata.com.

Upgrade to VAST Cluster 5.5.0 is supported from the following releases:

5.4.4 - 5.4.4-SP6
5.4.3 - 5.4.3-SP5
5.4.2
5.4.1 - 5.4.1-SP5
5.4.0 - 5.4.0-SP3

Note that direct upgrade may not be supported from hotfix builds. Consult VAST Support regarding upgrade if your cluster is running a hotfix build.

Note: VAST Cluster 5.5.0 is not compatible with VAST CSI 2.6.3 or earlier. Upgrade VAST CSI to 2.7.0 or later before upgrading to VAST Cluster 5.5.0.

For scale-related information, see VAST Cluster Scale Guidelines.

New Features

INSTALL AND UPGRADE
- Redesign of VAST Cluster Install
QUOTAS
- Usable Capacity-Based Quotas
QUALITY OF SERVICE (QOS)
- Quality of Service (QoS) for Block Volumes
NFS
- Support for NFSv4.2 Extended Attributes
- mTLS Authentication for NFS and Certificate-Based Tenant Identification
S3
- Support for Cross-Origin Resource Sharing (CORS)
EVENT PUBLISHING
- mTLS Authentication for VAST Event Broker
VAST DATABASE
VAST DATAENGINE
REPLICATION
- Asynchronous Replication of Block Subsystems and Volumes
PLATFORM AND CONTROL
- DBox Removal

Redesign of VAST Cluster Install

The VAST Cluster Install utility has been redesigned to improve user experience and provide for more robust deployment process. Among enhancements are extended pre-install validations with a user-friendly interface to list and resolve any issues found, improved checkpoints, and automatic skip and retry for components where the deployment cannot succeed for some reason.

The utility walks you through five basic steps, each represented with a corresponding step:

General Cluster Configuration - Specify if the cluster is an EBox or a combination of CBoxes and DBoxes, enter the cluster name and license, set up management IP allocation and B2B IPMI, enable HA and resiliency features, enable encryption, define racks. Click the Discover Boxes button to start the cluster node discovery process.
Infrastructure Mapping - The utility runs a discovery process and presents an inventory of discovered boxes and nodes for you to map them to the racks. The mapping can be done automatically based on a JSON inventory file.
Network Configurations - Here you set cluster networking parameters for both internal and external networks.
Advanced Settings - This step provides additional settings, including cluster's callhome configuration and more networking parameters.
Installation - This is where you validate the configurations made and start the installation process.

Installation settings presented in each step can also be imported or exported in JSON format.

Most of the steps offer an ability to validate the settings you've made before proceeding. When you reach the final step, you can run an overall pre-install validation, during which numerous checks are made to ensure that the requested configuration is consistent and can be safely implemented during the deployment phase.

During final pre-install validation and actual installation phases, you're presented with a detailed list of tasks being performed with their current status. The utility also offers a listing of issues that can cause a task to fail. For each issue, there is detailed information of the issue origin, severity and suggested corrective action. Thanks to the deployment checkpointing mechanism, you can pause the installation process to resolve the issues and resume it when ready.

For more information on VAST Cluster Install, see VAST Cluster Software Installation Guide.

Usable Capacity-Based Quotas

VAST Cluster 5.5.0 offers an ability to set quota limits based on the usable capacity. Prior to this release, the quota limits were set based on logical capacity only.

The capacity type is selected during quota creation and cannot be altered for an existing quota. Setting both logical and usable capacity limits within a single quota is not allowed. After selecting the capacity type for a quota, all of the limits defined in this quota will use the selected capacity type.

For the purpose of quota accounting, usable capacity is determined as follows:

Usable Capacity = Logical Capacity / Data Reduction Ratio (DRR)

For example, writing 200GB of data to a quota-limited directory with a DRR of 2:1 consumes 100GB of the directory's quota limit.

Quota consumption of a usable capacity-based quota may change over time due to changes in the cluster's DRR, without actual changes in files or directories.

The following user controls have been added:

In VAST Web UI, the capacity type selection options, Logical capacity and Usable capacity, in quota settings (Element Store -> Quotas -> choose to create or edit a quota -> Quota Rules tab)
In VAST CLI, the --is-physical-quota option on the quota create command.
In VAST REST API:
- The is_physical_quota property of a quota
- The is_physical_quota parameter in a POST request to the /quotas/ endpoint

For more information about capacity-based quota limits, see VAST Cluster Administrator’s Guide.

Quality of Service (QoS) for Block Volumes

VAST Quality of Service (QoS) policies can now be applied to block volumes provisioned on the cluster. The following QoS capabilities are supported for block volumes:

Static limits (maximum, burst, credits) for read and/or write bandwidth and IOPS
Limits based on used capacity and provisioned capacity (read and/or write bandwidth and IOPS per GB)
Total limits

In addition, QoS limits set at the tenant or cluster level are also enforced on block operations, in the same way as for other supported access protocols.

Not that QoS for block volumes cannot be defined at the subsystem (view) level. You have to attach a QoS policy to specific volumes.

Note: When the VAST cluster is configured to apply QoS for block volumes, the clients must be configured to use round-robin or queue-depth load balancing.

The following user controls have been added to let you configure QoS for block volumes:

In VAST Web UI:
- A new Volume option in QoS policy settings (Element Store -> QoS -> choose to create a QoS policy) to specify that the policy is intended for block volumes. After you select this option, proceed to the new Volume tab where you can configure the policy.
- A new QoS policy field in volume settings (Element Store -> Volumes -> choose to create or edit a volume), which you can use to assign a volume QoS policy to the volume.
In VAST CLI:
- The new VOLUME value can be specified on the --policy-type option of the qospolicy create command.
- The --qos-policy-id option on the volume create and volume modify commands.
- The --detach-qos-policy option on the volume modify command.
In VAST REST API:
- QoS policies:
  - In GET requests to the /qospolicies/ and /qospolicies/<policy ID>/ endpoints, the policy_type of a QoS policy can be returned as VOLUME.
  - In POST requests to the /qospolicies/ endpoint, the policy_type parameter can be specified as VOLUME.
- Volumes:
  - A GET request to the /volumes/ and /volume/<volume ID>/ endpoints returns the ID and the name of the QoS policy attached to the volume(s).
  - The /volumes/ endpoint accepts the qos_policy__id parameter in GET requests that lets you filter the output by QoS policy ID.
  - A POST request to the /volumes/ endpoint can contain a qos_policy_id parameter that lets you associate the volume with the QoS policy.
  - The /volumes/<volume ID>/ endpoint accepts the qos_policy_id parameter in PATCH requests.

Note: Block storage QoS cannot be set by creating a view QoS policy for a Block-enabled view.

For more information about QoS, see VAST Cluster Administrator’s Guide. Note the client requirements when using QoS for block volumes.

The following limitations apply:

Using more than 100 volumes that have QoS limits applied to them may result in inaccurate performance capping.
Prioritization of a volume QoS policy over cluster QoS limits is not supported.
ORION-266523: For volumes that share the same subsystem, it is recommended that all of the subsystem volumes have the same QoS limits set, or all of them do not have a QoS policy assigned, to prevent possible host IO performance impact.

Support for NFSv4.2 Extended Attributes

VAST Cluster 5.5 adds support for NFSv4.2 extended attributes (xattr), allowing clients to use their own, user-defined metadata for files and directories.

An extended attribute is a key-value pair that is subject to certain naming rules, which may depend on the current view policy settings.

The following request types are supported: GETXATTR, SETXATTR, LISTXATTRS, REMOVEXATTR.

Operations that involve extended attributes require appropriate permissions: ACCESS4_XAREAD, ACCESS4_XAWRITE or ACCESS4_XALIST. The operations are subject to VAST Audit logging and appear in VAST Catalog under a new column named nfs4_xattrs, with the ability to add user-defined VAST Catalog columns for NFSv4.2 attributes.

Extended attributes are supported for regular files and directories. When extended attribute operations are performed via hardlinks or symlinks, the attributes are applied to the target file. Applying extended attributes to the symlink itself is not supported.

A prerequisite to using NFSv4.2 extended attributes is that the tenant has NFSv4.2 support enabled.

For more information about this feature, see VAST Cluster Administrator's Guide.

The following rules and limitations apply:

Extended attributes are supported for the NFSv4.2 protocol only.
The VAST cluster does not enforce any limitation on the maximum number of extended attributes per file. However, some clients may have such a limitation when listing extended attributes.

mTLS Authentication for NFS and Certificate-Based Tenant Identification

VAST Cluster 5.5.0 supports mTLS authentication of NFSv3 and NFSv4 connections, where in addition to the client authenticating the server before establishing a secure connection, the server must also authenticate the client using a CA certificate.

In multi-tenant environments, you can configure the VAST NFS server to distinguish between mTLS client certificates associated with particulars tenants, and thus to direct the IO to the intended tenant based on the mTLS certificate (in addition to using the client IP or destination virtual IP for that purpose). This improves security while enforcing tenant isolation and tenant-level authorization for NFS clients.

You can choose whether you want to enforce mTLS authentication per view by setting a flag in the view policy. If the enforcement flag is set, the cluster denies access for clients that do not present an appropriate client certificate.

The following user controls have been added for this feature:

In VAST Web UI:
- The new Authentication settings for mTLS pane in tenant settings (Element Store -> Tenants -> choose to create or edit a tenant -> Advanced Protocol Settings tab) lets you do the following:
  - Create an ID string that will be used to identify the tenant during mTLS authentication of NFS connections (in the mTLS identifier field).
  - Add a CA certificate and, optionally, add a CRL file (used for certificate revocation) for the tenant.
- The NFS/Kafka mTLS Auth certificate type that you can select in the Certificate for field in VMS settings (Settings -> Certificates) to enter a global mTLS certificate and set a tenant association parameter (stating which field in the client certificate will specify the tenant).
- The Enforce mTLS encryption option in view policy settings (Element Store -> View Policies -> choose to create or edit a view policy -> NFS tab).
In VAST CLI:
- A new set of tlscertificate * commands to create and manage TLS and mTLS certificates. To configure mTLS for NFS, specify --protocol NFS on the commands.
- The --mtls-identifier and --reset-mtls-identifier options on the tenant create and tenant modify commands
- The --nfs-enforce-mtls and --no-nfs-enforce-mtls options on the viewpolicy create and viewpolicy modify commands.
In VAST REST API:
- A new set of /tlscertificates/* endpoints to create and manage TLS and mTLS certificates
- A POST request to the /tenants/ endpoint and a PATCH request to the /tenants/<tenant ID>/ endpoint can contain the new mtls_identifier parameter.
- A GET request to the /tenants/ or /tenants/<tenant ID>/ endpoint returns the mtls_identifier field for the tenant.
- A POST request to the /viewpolicies/ endpoint and a PATCH request to the /viewpolicies/<policy ID>/ endpoint can contain the new nfs_enforce_mtls parameter.
- A GET request to the /viewpolicies/ or /viewpolicies/<policy ID>/ endpoint returns the nfs_enforce_mtls field for the view policy.

For more information about this feature, see VAST Cluster Administrator’s Guide.

The following requirements and limitations apply:

mTLS authentication for NFS requires TLS 1.3.
The VASTNFS driver is required to use mTLS certificate-based tenant identification for NFSv3. (NFSv4 does not require VASTNFS.)
When using mTLS certificate-based tenant identification for NFSv3, the client must mount with mountproto=tcp.
Revocation of an mTLS certificate by using CRL Distribution Points (CDP) is not supported.
When using NFSv3, if a parent view does not have TLS or mTLS enforced, the VAST cluster allows unencrypted access to any of its child views, regardless of whether the child view has TLS or mTLS enforcement configured on it.
For clients using multiple certificates for different NFS exports (certificate per mount), Linux kernel 6.17 or later is required.
ORION-332649: A CRL file can contain up to 100 certificates to be revoked.

The following known issues may be encountered:

ORION-353874: A Setting mtls certificate for protocol NFS failed: CA certificate chain is empty error occurs when trying to upload a trust chain to tenant's NFS mTLS settings (Element Store -> Tenants -> choose to create or edit a tenant -> Advanced Protocol Settings tab -> Authentication settings for mTLS pane) where the root CA is not the first in the chain.

Support for Cross-Origin Resource Sharing (CORS)

VAST Cluster 5.5.0 introduces support for S3 Cross-Origin Resource Sharing (CORS) - a browser-enforced mechanism that controls how resources of a certain origin (domain) can be used from outside of that origin.

To configure CORS on a bucket, you create a list of origins that are allowed to fetch resources from the bucket, and also specify the allowed HTTP methods and headers.

The following user controls have been added for this feature:

In VAST Web UI, the new CORS tab in view settings (Element Store -> Views -> choose to create or edit a view -> S3 tab). This tab provides a visual editor for you to create CORS rules for the bucket, and also lets you import CORS configuration.
In VAST CLI:
- Commands to manage CORS configurations: view create-s3cors-configuration, view delete-s3cors-configuration, view show-s3cors-configuration
In VAST REST API:
- A GET request to the /views/<view ID>/s3cors_configuration/ endpoint returns the CORS configuration for the view. A POST request to this endpoint lets you configure CORS rules for the view. A DELETE request deletes the CORS configuration for the view.

You can also manage CORS by sending PutBucketCors, DeleteBucketCors and GetBucketCors requests to VAST S3 API.

Managing CORS configuration for a bucket can be allowed or prohibited via identity or bucket policy using the s3:PutBucketCORS, s3:GetBucketCORS and s3:DeleteBucketCORS actions.

Note that the existing cluster-wide setting to enable or disable S3 CORS still applies, although is no longer recommended. Enabling it allows all origins to fetch resources from all buckets on all tenants, unless the bucket has a bucket-level CORS configuration that stipulates otherwise. Any bucket-level CORS configuration takes precedence over the cluster-wide setting.

For more information about this feature, see VAST Cluster Administrator’s Guide.

The following limitations apply:

ORION-366116: VAST implementation of CORS does not include unique IDs for CORS rules.
ORION-314829: VAST Cluster does not include the Access-Contol-Allow-Origin header in case of a failed PutObject request.

mTLS Authentication for VAST Event Broker

VAST Cluster 5.5.0 introduces support for mutual TLS (mTLS) authentication between VAST Event Broker and event consumers, allowing the server to authenticate clients connecting to a Kafka-enabled view on the VAST cluster.

mTLS authentication can be enabled or disabled per VAST Event Broker view. If mTLS authentication is enabled, connections without a proper CA certificate are rejected.

The CA certificate can be tenant-specific or global (cluster-wide, accepted by all tenants).

Note: With VAST Event Broker mTLS in place, VAST Event Broker views hosted on multiple tenants can use the same virtual IP pool, since determination of the correct tenant for each client request will be done based on the information in the CA certificate.

To manage CA certificates and other related settings:

In VAST Web UI:
- For tenant-level settings, use the new Authentication settings for mTLS pane in tenant settings (Element Store -> Tenants -> choose to create or edit a tenant -> Advanced Protocol Settings tab). This pane lets you upload the CA certificate and set the mTLS identifier for the tenant.
- For cluster-wide settings, use the Settings -> Certificates tab and select the Kafka certificate type.
In VAST CLI:
- Use tlscertificate * commands with --protocol KAFKA. Specify the ID of the tenant you need on the --tenant-id option.
- To set tenant's ID for mTLS, use the --mtls-identifier and --reset-mtls-identifier options on the tenant create and tenant modify commands.
In VAST REST API:
- A new set of /tlscertificates/* endpoints to create and manage TLS and mTLS certificates
- A POST request to the /tenants/ endpoint and a PATCH request to the /tenants/<tenant ID>/ endpoint can contain the new mtls_identifier parameter.
- A GET request to the /tenants/ or /tenants/<tenant ID>/ endpoint returns the mtls_identifier field for the tenant.

To enable mTLS authentication for a VAST Event Broker view:

In VAST Web UI, select the Require mTLS option in view settings (Element Store -> Views -> choose to create or edit a view -> select the Kafka protocol for the view -> Kafka tab -> Authentication Methods pane).
In VAST CLI, run the view create or view modify commands with the --kafka-encrypted-auth-mechanism option set to mTLS.
In VAST REST API, set the view's kafka_encrypted_auth_mechanism parameter to MTLS.

For more information about this feature, see VAST Cluster Administrator’s Guide.

Vector Indexing

VAST Cluster 5.5.0 supports vector indexing of VAST Database tables.

When you create a table with a vector index, the table data is arranged according to vector distances of a specific vector column, thus allowing for faster and more accurate similarity search that requires less compute resources.

When configuring the index, you must specify the distance metric used to calculate nearest neighbors. Supported metrics include Euclidean distance or Cosine inner product.

To enable vector indexing for a VAST Database table:

In VAST Web UI, enable a new capability named Vector indexing for the database table with a vector column (DataBase -> VAST Database -> choose to create a table -> Additional Capabilities) and in the subsequent page, select the vector column and the distance function, which can be Euclidean distance or Cosine inner product.
When using VAST Query Engine, submit CREATE TABLE with a VECTOR CLUSTER statement.

Related changes in the VAST REST API include:

The /tables/ endpoint accepts vector_index_column_name and vector_index_distance_metric parameters in a POST request.
The vector_index_column_name and vector_index_distance_metric parameters can also be used in PATCH requests to the /tables/add_columns/ endpoint.
The /tables/show/ endpoint returns vector_index_column_name and vector_index_distance_metric in response to GET requests.

Note that while updating the index after insertion may take some time to complete, newly inserted vectors are immediately queryable using nearest neighbor search and/or regular filtering.

For more information about this feature, see VAST Cluster Administrator’s Guide.

The following limitations apply:

Once vector indexing is enabled for a table, it cannot be disabled.
After a vector index is created, its distance function cannot be changed.
Only one vector column can be indexed per table.
The word 'vector' cannot be used as the vector column name.
Adding a vector index to an existing table is not supported.
Adding or deleting columns in an indexed table is not supported.
Vector indexing cannot be used together with sorted tables and partitioned tables.
VAST Connectors for Trino, Spark and Dremio do not support vector index-based search.

Table Partitions

VAST Cluster 5.5 supports table partitions.

Partitioning of database tables improves performance when querying and managing large data sets by providing an ability to operate on a single table partition instead of the entire table.

Table partitioning is supported with VAST Query Engine, as well as with Trino and Spark connectors. For example, to create a table where the rows are partitioned by the day (from event_time) and event type via Spark:

CREATE TABLE my_table (
    event_id int,
    event_time timestamp(6),
    event_type string
)
PARTITIONED BY (day(event_time), event_type)
ORDERED BY (event_time)

Table partitions are defined during table creation and cannot be disabled once the table is created.

You can specify up an ordered list of up to 4 column transformations and up to 4 sort-key columns (which may overlap with column transformations).

Among supported operations on partitioned tables are: create a partitioned table, list partitions within a table, insert into a partitioned table, import from a Parquet file, query data from a partitioned table, delete a partition.

Operations with partitions can be allowed or denied using the s3:TabularDeletePartitions action in identity policies.

The following user controls have been added for this feature:

In VAST Web UI, a new capability named Partitions is available per database table (DataBase -> VAST Database -> choose to create a table -> Additional Capabilities).
In VAST CLI, the --partitions-config option on the table create command.
In VAST REST API:
- The /tables/ endpoint accepts the partitions_config parameter in POST requests.
- The partitions_config parameter can also be used in PATCH requests to the /tables/add_columns/ endpoint.
- A GET request to the /tables/show/ endpoint returns the partitions_config for the table.

For more information about this feature, see VAST Cluster Administrator’s Guide.

The following limitations apply:

Partitions cannot be created based on nested columns (list, map, struct column types).
Columns used to create partitions can only be updated if the update does not affect existing partitioning.
Columns used to create partitions cannot be dropped once the partition exists.
Table partitions cannot be used together with the following database features:
- Vector indexing
- Semi-sorted projections
- User-defined row IDs
Sorting can be used on partitioned tables, but the sorting order can only be defined at table creation. Existing partitioned tables cannot be sorted.
It is not allowed to use the same column both for sorting and partitioning.
ORION-302884: Quotas on partitioned tables are not supported.
Replication of partitioned tables is not supported.
VAST Query Engine does not support table partitions.

Blob Expansion: Transforming Event JSON into a VAST Database Table

VAST Cluster 5.5.0 offers an ability to transform events within an event topic from their JSON format to tabular format and store them as rows within an additional VAST Database table, which you can query in the same way as other database tables. This feature is referred to as blob expansion. The expansion is performed synchronously to adding the event into its event topic, using the same VAST Database transaction.

You enable and configure blob expansion per event topic. When configuring blob expansion, you specify the target schema and name of the target table (to which events records are expanded), the names and types for the target table columns, and optionally set other expansion parameters.

Note that in addition to the columns you defined, the expansion table will also contain extra columns, such as those used to track any issues that may occur during event record parsing.

Blob expansion requires the following permissions in the identity or bucket policy: s3:TabularCreateBlobExpansion, s3:TabularAlterBlobExpansion, s3:TabularDropBlobExpansion, s3:TabularGetBlobExpansion.

The following user controls have been added for this feature:

In VAST Web UI:
- The new Add Blob Expansion option for an event topic (DataBase -> VAST Database -> navigate to the event topic) opens a dialog where you can configure blob expansion for the topic.
- To alter an existing blob expansion, open the event topic menu (in the top-right corner of the event topic page) and choose Blob Expansion. You can also choose to remove blob expansion from the topic.
In VAST CLI, the blobexpansion create, blobexpansion show, blobexpansion add-columns, blobexpansion drop-columns, blobexpansion delete commands.
In VAST REST API, a new set of endpoints:
- A POST request to the /blobexpansion/ endpoint creates a new blob expansion.
- A GET request to the /blobexpansions/show/ endpoint returns blob expansion details.
- To add or drop columns in existing blob expansion, send PATCH requests to the /blobexpansions/add_columns/ and /blobexpansions/drop_columns/ endpoints respectively.
- To remove blob expansion, send a DELETE request to the /blobexpansions/delete/ endpoint.

For more information about this feature, see VAST Cluster Administrator’s Guide.

The following limitations apply:

Only one expansion table can be defined for a given column in the event topic.
The expansion target table must be created (without columns) before you start configuring blob expansion for the table being expanded.
Renaming or removing target table columns that are included in blob expansion configuration, is not allowed.
Updates/deletes to the source columns do not affect the target table.
Renaming of the source table is not allowed.

VAST Compute Clusters

With VAST Cluster 5.5.0, you can deploy your applications using an internally managed compute cluster that runs on VAST CNodes. This eliminates the need to provide for an external Kubernetes cluster to be able to use VAST DataEngine.

A VAST cluster administrator can provision a compute cluster on VAST as follows:

In VAST GUI, go to Data Engine -> Compute Clusters, click the Create Compute Cluster button (at top right) and select Create VAST Compute Cluster in the menu. In the dialog that opens, follow the wizard to enter details for the new VAST compute cluster, such as the cluster name, certificates, descriptive tags, specify how much of the VAST cluster resources can be consumed by the compute cluster, and select the CNodes to host the compute cluster.
In VAST CLI, use the new set of computecluster * commands.
In VAST REST API, use the new /computecluster/ endpoints, as well as the /certificates/validate_compute_cluster_certificates/ endpoint.

After a compute cluster is created, it can be associated with one or more tenants.

A VAST DataEngine user will be able to select the VAST-internal compute cluster when creating their pipelines in VAST DataEngine.

For more information about this feature, see VAST Cluster Administrator’s Guide.

The following rules and limitations apply:

Each compute cluster requires at least three CNodes (because three server nodes are required to support HA). If more than 15 CNodes have been selected for a compute cluster, five master nodes will be allocated.

Note: The role of the node (server/worker) is determined automatically considering the lifecycle state of the nodes, and may change throughout the lifecycle of the VAST cluster (for example, role changes are expected in maintenance operations such as FRU and NDU). Depending on the number of active worker nodes, server nodes may also execute workload.
A CNode can be used for one compute cluster only.
ORION-348513: Compute clusters cannot be deployed in a environment with CNode port affinity (where the LEFT port and the RIGHT port of a CNode can be on different networks).
Up to 32 VAST-internal compute clusters can be created per VAST cluster.
ORION-291582: VAST-managed compute clusters are not available in VAST on Cloud.
ORION-365247: VAST-managed compute clusters cannot be run on CNodes with L3 networking.
ORION-288369: VAST-managed compute clusters are not supported in environments with Layer 3 data networking.
ORION-371504: IPv6 addressing is not supported.
ORION-342312: There is no validation that the IP specified for a compute cluster node is free.
ORION-397536: VAST-managed compute clusters cannot be run on ENodes.

The following known issues may be encountered:

ORION-379170: The Tenant dropdown on the Data Engine -> Compute Clusters page may take a long time to load the list of tenants.

User-Defined Metrics

In addition to VAST-supplied metrics for functions and pipelines, VAST DataEngine 5.5.0 offers support for user-defined metrics, so that DataEngine users can define, produce and visualize their own OpenTelemetry-compatible metrics.

The following additions have been made for this feature:

In VAST DataEngine UI, a new dashboard named Custom Metrics has been added to VAST DataEngine UI, where you can set up and display graphs based on selected counters and time ranges
In VAST DataEngine CLI, a new metrics get command to retrieve metrics
In VAST DataEngine Runtime SDK, the new context.meter interface for metrics creation.

For more information about this feature, see VAST DataEngine User Guide.

NFS-Based Triggers

Starting with VAST Cluster 5.5.0, VAST DataEngine users can create triggers based on events occurring to elements in VAST NFS file storage. (Prior to this release, only S3 bucket events could be used.)

To create an NFS-based trigger in VAST DataEngine UI, select NFS in the Source Type field when creating a trigger (Manage Elements -> Triggers -> choose to create or edit a trigger).

The VAST REST API offers a new endpoint, /views/<view ID>/nfs4_triggers/, that lets you view and manage NFS triggers registered for a particular view. In addition, a GET request to the /views/ or /views/<view ID>/ endpoint returns the has_nfs4_triggers parameter that indicates whether NFS triggers are enabled for the view(s).

The following table shows the mapping between supported NFS operations and event types in trigger configuration:

Trigger Event Type	NFS Operation
Element Created	Create a file
Element Deleted	Delete a file

For more information on managing triggers, see VAST DataEngine User Guide.

The following requirements and limitations apply:

Only regular files can be the source for trigger events. Directories, symlinks and other special file types are not supported.
Only NFSv4 is supported. Events resulting from NFSv3 or SMB operations on the same source view would not be used for the trigger.
The source view must have NFSv4 and S3 Bucket protocols enabled.
NFSv4 file delegations must be disabled on the view configured as the trigger's event source view.
Events occurring to elements on nested views of the source view, cannot be used as the basis for VAST DataEngine triggers. For example, if an NFS-based trigger is configured on a source view at /abc/, an event on its nested view /abc/xyz/ will not trigger any action in VAST DataEngine.
If a file is renamed or deleted while it is being created (e.g. the file has been opened and written to, and then it got unlinked or removed), no create or delete event is produced.
ORION-320111: During an upgrade, the VAST cluster does not produce Element Deleted events for files subject to NFSv4 unlink and rename operations.

Event Batching

VAST DataEngine 5.5.0 supports event batching. Events can be accumulated into batches until a user-defined threshold is reached, and then the event handling function processes them in a single function call.

In VAST DataEngine UI, you configure batch settings in the new Batches pane of the pipeline visual editor (Pipeline Management -> Pipeline Management). Settings include the number of events to accumulate and the time to wait before processing.

With batched events, traces are available for a batch and per individual event. The VAST DataEngine UI provides a new filter named Batch Traces to display batch-level traces in the Pipeline Management -> Traces page.

For more information about this feature, see VAST DataEngine User Guide.

Conditional Triggers

With VAST DataEngine 5.5.0, you can programmatically control which downstream functions are triggered by a certain function.

In VAST DataEngine UI pipeline builder, you set a condition for a link between two functions by attaching a trigger label to it. The label is a key-value pair, for example: LoggingLevel:Debug. For the downstream function to run, the upstream function handler must set the specified value for the key. If multiple labels are attached, the upstream function handler must set all of the keys to their respective values.

Conditional trigger labels can be defined in the new Event Trigger Label pane in the right-hand panel of VAST DataEngine pipeline visual builder.

For more information about this feature, see VAST DataEngine User Guide.

Asynchronous Replication of Block Subsystems and Volumes

VAST Cluster 5.5.0 supports asynchronous replication of block subsystems and block volumes.

Subsystems created at the destination peer are associated with the default view policy for block views. If you want to use a different view policy, assign it manually at the destination peer.

The NQN of a subsystem created at the destination peer is different from that of the original subsystem.

Replicated block volumes can be mapped to hosts at the destination peer for read-only access.

The configuration workflow is similar to the one used for other types of protected paths, as described in the VAST Cluster Administrator's Guide.

The following block-specific rules and limitations apply:

If a replication group includes a replication stream that replicates block volumes:
- All replication peers in the group must be running VAST Cluster 5.5.0 or later.
- The replication group can include only asynchronous replication streams.
Block hosts and mappings between the volumes and the hosts are not replicated.
The default block view policy is to be created manually for the relevant tenant at the replication destination peer.
When using asynchronous replication of block volumes with ESXi, user intervention is required to unmap volumes from the ESXi host if they are no longer writeable (e.g. after a failover). Otherwise, volumes that are intended to be presented as read-only, are not being detected as such by ESXi.

DBox Removal

VAST Cluster 5.5.0 offers an ability to seamlessly remove an active DBox from a running cluster, with the data automatically moved to the remaining DBoxes. Prior to the removal, the VMS accomplishes validations to ensure that the reduced cluster can safely accommodate all of the data and meet the DBox HA, replication, balance, and RAID requirements (depending on which features are enabled on the cluster). As soon as the removal process starts, the cluster scales down to operate within the new capacity limits.

Note: Although the functionality is referred to as DBox removal, it is applicable to both DBoxes and EBoxes, as well as to VAST on Cloud VMs. An alternative name for the functionality is DBox decommission.

You can specify if you'd like to have the removed DBox(es) or EBox(es) powered off after the removal process completes, and also if you want to clean up the box-related configuration records from the VMS.

The following user controls have been added for this feature:

In VAST Web UI:
- The Remove DBox or Remove EBox context menu option for a box or boxes displayed in the Infrastructure -> DBoxes or EBoxes page
- The new Remove DBox or Remove EBox option in the Adding and Changing Boxes dialog (Infrastructure -> DBoxes or EBoxes -> Manage Infrastructure button on the right)
In VAST CLI, the dbox decommission, ebox decommission and vm decommission commands
In VAST REST API, new sets of endpoints: /dboxes/decommission/* and /eboxes/decommission/*, as well as the new /virtual-machines/decommission/ endpoint.

DBox removal can be performed by users with the Delete permission on the Hardware realm.

You can monitor the status of a DBox removal process in VAST Web UI Activities page or by clicking the Activities button on top right of the Infrastructure -> DBoxes or EBoxes page. VAST audit logs and event logs provide a detailed record of each process step.

For more information about this feature, see VAST Cluster’s Administrator’s Guide.

The following rules and limitations apply:

A DBox removal process can only be started after metadata rewrite is complete (metadata rewrite is required when upgrading to VAST Cluster 5.5). See Limitations for metadata rewrite.
DBox removals reducing the cluster size below the initial installation size may not be allowed.
Once the DBox removal process has started, it cannot be stopped.
Only one DBox removal process can be run at a time. The process can involve one or more DBoxes.
DBox removal cannot be performed at the same time when DBox migration is running (even if it is for a different DBox).
DBox removal cannot be performed at the same time when DBox replacement is running.
During DBox removal, it is not allowed to start a DBox expansion process. Adding or removing CBoxes is allowed.
During DBox removal, the Write Buffers on Flash functionality is disabled.
DBox removal is not supported after a Write Buffer RAID rewrite.
If the cluster has DBox HA enabled, removing a DBox is not allowed if it would reduce the cluster below the minimum number of DBoxes required for DBox HA.
If the cluster has rack resiliency enabled, the DBoxes to be removed must be balanced across all failure domains so that the removal does not impact failure domain balancing. Alternatively, the DBoxes to be removed must constitute a single failure domain.

Enhancements

Install and Upgrade

Added an ability to skip failed DBoxes during a VAST Cluster Install procedure. The skipped DBoxes can later be added through cluster expansion.
VAST Cluster Install lets you configure the out-of-band gateway and CIDR per rack.
VAST Cluster Install lets you set up a template for generating cluster node names based on a number of variables, such as cluster name, rack name, box type and so on. Prior to this change, you could only set a user-supplied prefix when generating node names.
Added the following user controls to help manage VMS task issues (in addition to VAST Cluster Install capabilities):
In VAST CLI, the new issue list command
In VAST REST API, the new /issues/ and /issues/pre_install-validations/ endpoints.
ORION-375379: Added an ability to include or skip upgrade of internal VAST compute cluster instances deployed across the CNodes when running a cluster upgrade procedure:
In VAST CLI, use the --compute-upgrade on the cluster upgrade command.
In VAST REST API, use the compute_upgrade parameter in a PATCH request to the /clusters/<ID>/upgrade_without_file/ endpoint.
ORION-284784: Added a validation to prevent filling the Subnet field of VAST Cluster Install with a value that ends with a dot (for example, 10.10.).
ORION-283464: Made updates to preserve the order in which the IPs are entered in the VAST Cluster Install fields so that they do not get sorted during further configuration processing.
ORION-251554: Introduced changes in the structure of VAST Cluster installation bundles to reduce the overall bundle size.

Networking

Added user controls to manage allocation of virtual IP pools and DNS in VoC environments:
In VAST CLI, the vippool allocate, vippool reallocate, dns allocate commands
In VAST REST API, the /vippools/allocate/, /vippools/<pool ID>/reallocate/ , /dns/allocate/ endpoints.

Element Store

VAST Cluster 5.5.0 introduces default block view policies with the new Block security flavor.

The following user controls have been added:
- In VAST Web UI, the Block checkbox under Set Default View Policies in tenant settings (Element Store -> Tenants -> choose to create or edit a tenant -> Advanced Protocol Settings tab).
- In VAST CLI, the enable-block-default-policy and disable-block-default-policy options on the viewpolicy create and viewpolicy modify commands.
- In VAST REST API, the is_block_default_policy parameter for view policies.
ORION-286311: Introduced optimizations to improve performance when handling workloads that sequentially create multiple files under the same directory (such as untar or git clone).
ORION-280834: Added user controls to return the total number of existing views for a tenant along with the maximum number of views allowed:
- In VAST CLI, run the tenant views-count command.
- In VAST REST API, send a GET request to the /tenants/<tenant ID>/views_count/ endpoint.
ORION-260411: VAST Cluster 5.5.0 introduces a new security flavor, Block, intended for views that have the block access protocol enabled. You can select the new flavor in a view policy, similar to other security flavors:
- In VAST Web UI, choose the Block option in the Security flavor field of view policy settings (Element Store -> View Policies -> choose to create or edit a view policy -> General tab)
- In VAST CLI, specify --flavor BLOCK on the viewpolicy create or viewpolicy modify command.
- In VAST REST API, the view policy's flavor parameter can be set to BLOCK.
The new flavor is used in the default block view policy that is automatically assigned to replicated block views at the destination peer.
ORION-235925: Increased the maximum allowed number of views (NFSv3, NFSv4, SMB, S3, VAST Database) to 153600.

Note: Due to limitations of the showmount utility, the showmount -e command may fail with a segmentation fault (core dumped) error on an attempt to list a very large number of views (more than 32K).

Quality of Service (QoS)

Added support of QoS for NFS over RDMA.
Added the ability to set a maximum read bandwidth for the entire cluster. To do so:
- In VAST Web UI, go to the Settings -> Cluster -> General Cluster Setup and Actions -> Global BW Limits pane, select Set Manually and enter the bandwidth in the Max Read BW field.
- In VAST CLI, use the --max-cluster-read-bw-mb option on the cluster modify command.
- In VAST REST API, include the max_cluster_read_bw_mb parameter in a PATCH request to the /clusters/<cluster ID>/ endpoint.
With VAST Cluster 5.5.0, block volumes are subject to per-tenant QoS limits, and also to cluster-wide QoS limits. These limits are configured in tenant and/or cluster settings without the need to create a QoS policy.
ORION-290798: Added a validation to prevent users from configuring QoS maximum limits that are lower than than 100Mb/s or 1000IOPS.
ORION-257434: Added a metric that shows IO latency due to tenant-level QoS limits (qos_tenant_latency_us).

Protocols

Enhanced the functionality for listing open files and open file handles:
- In VAST CLI:
  - The openfilesquery list-open-file-handles command features new filter options to filter the output by client IPs (--client-ip-startswith, --client-ip-subnet), access protocol (--protocol), username (--username-contains), and also by the presence of locks (--has-locks) or leases (--has-lease).
  - The openfilesquery list-open-files command provides new filter options:
    - --has-locks to list only those open files that have locks on them
    - --path-contains to filter open files by path
  - The new openfilesquery list command offers new options to filter the output by path (--path-prefix-contains), query name (--name-contains), tenant ID (--tenant-id), query created date and time (--created-gte, --created-lte), and query status (--state).
  - The new openfilesquery refresh command lets you rerun a existing open files query.
  - The openfilesquery query-open-file-handles command has been deprecated.
- In VAST REST API:
  - The /openfilehandles/ endpoint accepts the client_ip__startswith, client_ip_subnet, has_locks, has_lease, usernames__icontains, protocol filter options in GET requests.
  - The /openfiles/ endpoint accepts the path__icontains and has__locks filters in GET requests.
  - The /openfilesqueries/ endpoint accepts the path_prefix__icontains, name__icontains, tenant__id, created__gte, created__lte, state filter options in GET requests.
ORION-234868: Improved performance of XCOPY operations for the block protocol (e.g. for VM cloning), and also for the SMB server-side copying.

NFSv3

ORION-316673: Added support for NFSv3 mounts using the NFS alias set for a view. Prior to this change, only mounting by view path was allowed.
ORION-306984: This release reintroduces functionality for handling of NFSv3 aliases so that after mounting a view with an alias through NFSv3, NFS export listing commands like showmount -e would show both the alias and the export path.

NFSv4

VAST Cluster 5.5.0 lets you configure NFSv4 file delegations both per tenant and per view. Settings made at the tenant level can be used as defaults for all the views within the tenant. Settings made per view override the tenant-level settings for that view (with the exception of the setting that disables NFSv4 file delegations for the tenant and its views).

The following user controls have been added for this purpose:
- For a tenant:
  - In VAST Web UI, the Access Delegation Policy pane in tenant settings (Element Store -> Tenants -> choose to create or edit a tenant -> Advanced Protocol Settings tab) offers two new options, Enable tenant level delegations and Disable tenant level delegations, and also an option to avoid setting the delegation type (read and/or write) at the tenant level.
  - In VAST CLI, the --allowed-delegations option on the tenant create and tenant modify command accepts a new value, DISABLED, that disables both tenant-level and view-level configuration of NFSv4 delegations.
  - In VAST REST API:
    - A new value of DISABLED can be specified on the allowed_delegations parameter in POST requests to the /tenants/ endpoint and in PATCH requests to the /tenants/<tenant ID>/ endpoint.
    - The DISABLED value can be returned by GET requests to the /tenants/ and /tenants/<tenant ID>/ endpoints.
- For a view:
  - In VAST Web UI, the new NFS -> Access Delegation Policy tab in view settings (Element Store -> Views -> choose to create or edit a view) lets you accept or reject tenant-level configuration for the view, and also enable read and/or write delegations for this particular view.
  - In VAST CLI, the new --allowed-delegations option on the view create and view modify commands.
  - In VAST REST API:
    - A POST request to the /views/ endpoint and a PATCH request to the /views/<view ID>/ endpoint can contain the allowed_delegations parameter that sets delegation configuration for the view.
    - GET requests to the /views/ and /views/<view ID>/ endpoints return the view's delegation configuration in the allowed_delegations field.
Following an upgrade to version 5.5:
- If a tenant had the allowed delegations option set to NONE, the value is automatically changed to DISABLED.
- New tenants are by default created with allowed delegations set to READ_WRITE.
- All new views are by default configured to allow tenant-level delegations.
ORION-221143: Added support for using NFSv4 delegations with RDMA.

SMB

ORION-282888: Added a column named SMB Encryption to the Element Store -> Views page. This column shows the status and mode of SMB 3.0 encryption (Available/Desired/Required/Off).
ORION-249933: Added performance optimizations for SMB Server-Side Copy.

S3

Updated handling of HeadBucket requests so that the cluster responses contain the x-amz-bucket-region header set to the vast region.
ORION-254573: Added support for the If-Match HTTP header in GetObject and HeadObject requests. The request succeeds only if the object's ETag matches the ETag provided in the request. Otherwise, a PreconditionFailed error is returned.

Block

ORION-288363: Added support for subsystem names that contain a space (in VAST Web UI: Element Store -> Views -> choose to create a view -> Block tab -> Subsystem name field).
ORION-225395: Made updates to ensure that after mapping a snapshot to a block host, the mapped volume is listed as such for the host in both VAST Web UI and VAST CLI.
ORION-215384, ORION-359297: The Element Store -> Volumes page of VAST Web UI features new columns:
- Full Path to show the full path to the volume (including the subsystem name).
- Creation Time to indicate the volume creation date and time.

Event Publishing

Added an ability to use a single virtual IP pool for VAST Event Broker views belonging to different tenants, on condition that each participating view has mTLS authentication enabled. Determination of the correct tenant for each client request will be done based on the information in the client certificate.

This change lifts the requirement for the virtual IP pool to belong to the same VAST tenant as the VAST Event Broker view.
Added the following user controls that lets you fine-tune retention and compaction of messages in an event topic:
- Minimum time to keep an uncompacted message in the partition log:
  - In VAST Web UI, the Minimum compaction lag option in event topic settings (DataBase -> VAST Database -> go to a database with event topics -> choose to create or edit a topic)
  - In VAST CLI, the --min-compaction-lag-ms option on the ``topic createortopic modify` commands
  - In VAST REST API, the min_compaction_lag_ms parameter in POST requests to the /topics/ endpoint and in PATCH requests to the /topics/<ID>/ endpoint.
- The period during which tombstone records can be kept in a topic for which compaction is enabled:
  - In VAST Web UI, the Tombstone Retention Period option in event topic settings (DataBase -> VAST Database -> go to a database with event topics -> choose to create or edit a topic)
  - In VAST CLI, the --delete-retention-ms option on the topic create or topic modify commands
  - In VAST REST API, the delete_retention_ms parameter in POST requests to the /topics/ endpoint and in PATCH requests to the /topics/<ID>/ endpoint.
- The default cleanup policy for segments beyond the retention period (delete or compact):
  - In VAST CLI, the --cleanup-policy option on the topic create or topic modify commands
  - In VAST REST API, the cleanup_policy parameter in POST requests to the /topics/ endpoint and in PATCH requests to the /topics/<ID>/ endpoint.
- The maximum allowed size for a message:
  - In VAST Web UI, the **Max size(( option in event topic settings (DataBase -> VAST Database -> go to a database with event topics -> choose to create or edit a topic -> Advanced options)
  - In VAST REST API, the max_message_bytes parameter in POST requests to the /topics/ endpoint and in PATCH requests to the /topics/<ID>/ endpoint.

VAST DataBase

Added support for using Spark to enable table sorting.
Increased the maximum cell size (including the metadata) from 126KB to 5MB. The new limit is not supported for semi-sorted projections.
For fixed size binary columns, the maximum cell size is 64K.
For Kafka topics, the maximum cell size is 1GB.
The maximum allowed statement size is 5MB.

VAST Query Engine supports DDL commands used to create a schema, as well as manage tables and columns:

  CREATE [IF NOT EXIST] SCHEMA
  CREATE [OR REPLACE | IF NOT EXIST] TABLE
  CREATE [OR REPLACE | IF NOT EXIST] TABLE ... ( … CONSTRAINT ... SORT KEY (...))
  DROP TABLE
  DESCRIBE [TABLE]
  ALTER TABLE ... RENAME TO ...
  ALTER TABLE ... RENAME COLUMN ... TO 
  ALTER TABLE ... ADD [COLUMN] 
  ALTER TABLE ... DROP [COLUMN]
  ALTER TABLE ... ADD CONSTRAINT ... SORT KEY (...)
  CREATE VIEW ... AS ...
  DROP VIEW ...

Use of DDL commands requires the s3:Tabular permission.

The following limitation applies:

The supported set of DDL commands does not include commands for managing projections or partitions.

VAST Query Engine supports DLM commands used to insert, update and delete data, and also to import data from a Parquet file:
```
  INSERT INTO ... VALUES (...)
  INSERT INTO ... SELECT ...
  INSERT INTO ... (<colnames> "<filepath>") VALUES (<values>)
  UPDATE ... SET ... WHERE ...
  DELETE FROM ... WHERE ...
```
Note: For complete command syntax, see VAST Cluster Administrator's Guide.

Use of DLM commands requires the s3:Tabular permission.
VAST Query Engine supports SQL aggregate functions, such as SUM, MIN, MAX, AVG and so on, including support for the FILTER clause. The following clauses are not supported with aggregate functions: DISTINCT, ORDER BY, GROUP BY.
ORION-306122: Made updates to ensure that VAST Query Engine supports sorted tables without any caveats.
ORION-300539: Added support of VAST Database Row/Column-Level Security for Spark query engines.
ORION-297589: Added support for Trino 478 query engines.
ORION-237998: The following limitation has been removed:
- When calculating the sorting status, a constant value within the sorted value range is returned for tables that contain less than 5 million rows.

VAST Catalog

You can add user-defined columns for NFSv4.2 extended attributes. To view and configure these columns:
- In VAST Web UI, go to Settings -> VAST Catalog -> User-defined attributes. Select NFSv4 xattrs in the Type field and complete other fields as needed.
- In VAST CLI, use --column-type nfs4_xattr on the vastcatalogindexedcolumn * commands.
- In VAST REST API:
  - The column_type parameter in PATCH requests to the /bigcatalogindexedcolumns/add/ endpoint can contain a new value, nfs4_xattr.
  - A column type of nfs4_xattr can be returned by GET requests to the /bigcatalogindexedcolumns/ endpoint.

VAST DataEngine

In VAST Web UI, the Data Engine -> Kubernetes Clusters page has been renamed to Compute Clusters. The page displays a list of both VAST-internal and external Kubernetes clusters configured for use with VAST DataEngine.
ORION-307991: Added support for pipeline and function filters to /dataengine/function-revisions/ and /dataengine/pipeline-revisions/ endpoints.
ORION-294787: VAST DataEngine UI dashboard features the following new graphs:
- Total Events And Failures
- Function Invocation
- Active Pipelines
- Failing Pipelines
- Slowest Pipelines
  In addition, the dashboard lets you build and display a graph based on custom metrics of your choice.

Replication

The option to enable or disable bucket replication for a cluster (in VAST Web UI: Settings -> S3 tab -> Bucket Replication option) has been deprecated.

Starting with release 5.5.0, bucket replication is always disabled at the cluster level for newly installed clusters.

For clusters upgraded to 5.5.0:
- If the cluster had bucket replication enabled, all existing protected paths will have bucket replication enabled after the upgrade.
- If the cluster had bucket replication disabled, all existing protected paths will have bucket replication disabled.
The options to enable or disable bucket DB replication have been deprecated.
Added an ability to enable or disable bucket replication per protected path or replication stream. To do so:
- In VAST Web UI, use the new Enable bucket replication option that appears when you're adding the remote replication peer in protected path or DataSpace setup.
- In VAST CLI, use the --enable-bucket-replication and --disable-bucket-replication options on the protectedpath create, protectedpath add-stream, replicationstream create, globalsnapshotclone create commands.
- In VAST REST API, a POST request to the /protectedpaths/ endpoint can contain a new parameter, bucket_replication_enabled. The new view_replication_enabled parameter (used for the same purpose) can also be included on PATCH requests to the /protectedpaths/<id>/add_stream and POST requests to the /globalsnapstreams/ endpoint
Note that the new option can only be applied at the creation time. You cannot enable or disable bucket replication per path/stream after the path or stream has been created.

VAST DataSpace

Added support for use of the S3 Native security flavor on NFSv3 access to a view configured as a Global Access satellite path (at a destination peer). Prior to this change, only NFS flavor was supported. This lifts the restriction on enabling both NFSv3 and S3 protocols on a satellite view.
ORION-265298: Added dataspace task-list and dataspace task-delete VAST CLI commands that you can use to view and delete VAST DataSpace tasks as needed (for example, if a cluster deletion task got stuck in a pending state).

VAST on Cloud (VoC)

Added an ability to withstand platform-initiated periodic maintenance of VMs used in VoC clusters. When a notice on upcoming maintenance is received for a VM, the cluster migrates all data from it to a new VM and terminates the old one.
Added an ability to downscale a running VoC on GCP cluster by reducing the number of VMs. The process involves checking that there is enough space to accommodate the data on the remaining VMs, copying the data, and terminating the VMs that are not needed. Note that the cluster cannot be reduced below the minimum allowed cluster size.

Authentication and Authorization

ORION-268096: Increased the maximum number of identity policies per user from 20 to 120.

VMS

Improved the mechanism of collecting, aggregating and reporting cluster metrics to ensure that these processes meet the requirements of larger clusters.
The following metrics have been deprecated:
- rpc_get_fiber_latency
- rpc_get_fiber_and_read_latency
- s3_mpu_parts_number
- s3_mpu_part_size
- s3_mpu_file_size
Added CLI capabilities to manage the settings for automatic logout from VAST Web UI after idle time:
- The vms modify_idle_timeout_settings command lets you set the idle timeout.
- The --idle-timeout-settings filter on the vms show command lets you view the current idle timeout settings.
ORION-365744: Added a metric named Block Volume Offloaded IOs Bandwidth that shows the bandwidth for offloaded block operations.
ORION-278435: Introduced updates to help avoid node container restarts by adopting a new mechanism for phasing out memory sections prior to DNode deactivation.
ORION-253054: Added metrics that provide the total number of NFSv3 NLM read/write lock requests:
- read_lock_request_cnt
- read_unlock_request_cnt
- write_lock_request_cnt
- write_unlock_request_cnt
ORION-239505: Expanded syslog setup with a new option that enables or disables logging to the /var/logs/secure log:
- In VAST Web UI, the Enable Secure Log Audit option in Settings -> Notifications -> Syslog Setup
- In VAST CLI, the --syslog-enable-secure-audit and --syslog-disable-secure-audit option on the eventdefinitionconfig modify command.
- In VAST REST API:
  - POST requests to the /eventdefinitionconfigs/ endpoint and PATCH requests to the /eventdefinitionconfigs/<config ID>/ endpoint can contain the new syslog_secure_audit parameter.
  - A GET request to the /eventdefinitionconfigs/ or /eventdefinitionconfigs/<config ID>/ endpoint returns the new syslog_secure_audit field.

VAST Web UI

ORION-368089: Added an option to update DataEngine Kafka CA certificate to the right-click menu for a tenant in the Element Store -> Tenants page.
ORION-362975: The identity policy visual builder (User Management -> Identity Policies -> choose to create or edit an identity policy) now includes a new action category for custom statements under Available Actions -> Database -> Query Engine that lists supported VAST Query Engine actions.
ORION-102235: The Physical Capacity column in the Element Store -> Views page has been renamed to Used Capacity.

VAST CLI

The --path-length option on the viewpolicy create and viewpolicy modify commands has been renamed to --max-name-length. This option affects not only the length of a file path component, but also the length of NFSv4.2 extended attribute names and values.
The --path and --create-dir options on the view modify command have been deprecated.
The --local-provider-id option on multiple commands has been renamed to --vast-provider-id.
The --kdc and --kadmin-servers options on the kerberos create command are no longer required options.
Renamed some value keywords for the --capabilities and --source-member-capabilities options on the protectedpath create, protectedpath modify, protectedpath add-stream, protectedpath modify-member commands:
- STARED_GLOBAL_NAMESPACE -> GLOBAL_ACCESS
- REPLICATION_AND_GN -> ASYNC_REPLICATION_AND_GLOBAL_ACCESS
ORION-269350: The --protection-policy-name option on the protectedpath list command has been renamed to --replication-policy-name.

VAST REST API

The transport_mode parameter for the /nativereplicationremotetargets/ endpoint has been deprecated.

Callhome and Support

Added an ability to collect traces split into slices according to a user-supplied time value. For example, when a bundle is requested for 6 hours with the split time set to 5 minutes, VMS will create multiple 5-minute bundles and send them one by one.

To set the split time:
- In VAST Web UI, use the new Split time field in support bundle settings (Support -> Support Bundle -> choose to create a support bundle -> Advanced tab)
- In VAST CLI, specify the --split-time option on the supportbundle create command.
- In VAST REST API, use the split_time parameter in POST requests to the /supportbundles/ endpoint.
Note that setting this split time for a support bundle requires the Delete after sent flag is set for the bundle.
Added more filters to select the nodes from which to collect support bundles:
- In VAST CLI, the new --cnode-names, --dnode-names, --dbox-ids, --dbox-names, --dtray-ids, --dtray-names, --all-cnodes, --all-dnodes options on the supportbundle create command.
- In VAST REST API, new parameters in a POST request to the /supportbundles/ endpoint: cnode_names, dnode_names, dbox_ids, dbox_names, dtray_ids, dtray_names, all_cnodes, all_dnodes.
Added user controls to modify a support bundle:
- In VAST CLI, the new supportbundle modify command
- In VAST REST API, the /supportbundles/<bundle ID>/ endpoint accepts PATCH requests.
You can limit the scope of support bundle collection to include only those CNodes that are hosting or have been hosted (during the bundle's time range) specific virtual IPs in the virtual IP pool selected for the bundle. To supply the virtual IPs of interest:
- In VAST CLI, use the new --vip-ids option on the supportbundle create command.
- In VAST REST API, use the new vip_ids parameter in a POST request to the /supportbundles/ endpoint.
The following user controls have been deprecated:
- In VAST Web UI, the Luna max frequency (hours) option in callhome settings
- In VAST CLI, the --luna-on-alarm-interval option on the callhomeconfig modify command
- In VAST REST API, the luna_on_alarm_interval parameter for the /callhomeconfigs/<config ID>/ endpoint.
The option to delete the support bundle after sending, Delete after send, is now available in support bundle settings in VAST Web UI.
VAST Cluster 5.5.0 introduces a queuing mechanism for support bundles. When you create a support bundle, it is placed at the end of the queue with a status of Queued. Thus you can create a new bundle without having to wait till the previous one completes.
- If you want to check the position of your bundle in the queue in VAST Web UI, display the Position in Queue field in the Support Bundles page (Support -> Support Bundle).
- To view and manage the support bundle queue in VAST CLI, use the new commands: supportbundlequeue list and supportbundlequeue move.
- In VAST REST API:
  - Use the new set of /supportbundlequeue/* endpoints to manage support bundle queue.
```
- The bundle's position in the queue is returned in the new position_in_queue field in response to a GET request to the `/supportbundles/<bundle ID>` endpoint and to a POST request to the `/supportbundles/` endpoint.
```

Resolved Issues

Install and Upgrade

ORION-270481: Resolved an issue where an attempt to install a Cisco EBox could not proceed due to a Reset: Network dropped connection on reset error at the drive firmware downgrade step.

Cluster Expansion

ORION-284900: Resolved an issue that could occur during cluster expansion and cause CNode containers to restart with the assertion failed: (result < P::MAX_DBOXES_PER_SYSTEM) error.
ORION-147859: Enhanced the CBox and DBox expansion procedures to allow running the expansion without having a management network in place. This enhancement eliminates the requirement to provide a management IP when adding new nodes to the box.

Replacements

ORION-265779: Made updates to the way VMS handles cluster internal networking reconfigurations during DTray replacement procedures to resolve an issue that could prevent a DTray replacement task from completing successfully.

Networking

ORION-95338: Reduced the time the cluster networking configuration script (configure_network.py) script waits for IPMI parameters to be set during network configuration on a Ceres platform.
ORION-327378: Made updates to avoid raising an assertion failed: (result || Globals::test_mode || Globals::loopback) alert in case an incorrect virtual IP configuration was specified when creating or modifying a virtual IP pool.
ORION-186050: Added a validation to ensure that IPv6 link-local addresses (fe80::) addresses are not accepted when creating a virtual IP pool.

Element Store

ORION-386887: Eliminated a race condition that could cause a newman did not find @newman_tid=<...> alert on the cluster.
ORION-365148: Resolved an issue that could cause the CNode container to restart with the assertion failed: (empty()) destroying a non empty list error.
ORION-336120: Eliminated an intermittent race condition that could cause the CNode container to restart with the assertion failed: (existing_phandle.is_valid()) found an existing link with no valid handle?! error when running synchronous replication.
ORION-333518: Resolved an issue that could cause occasional quarantining and denylisting of a file handle following a did not find a conflicting hash to update matching hash=0x2a matching aux hash=0x0 alert.
ORION-333075: Resolved an issue where a contention occurring when prefetching metadata in a Global Namespace setup could cause an ESTORE GN_MD_PREFETCHER automatic denylist alert on the cluster.
ORION-269853: Improved internal data caching routines to avoid a flow that could cause multiple CNode container restarts and increased write latency.

Quotas

ORION-320022: The Quotas page in VAST Web UI (Element Store -> Quotas) has a new column that displays quota groups associated with a quota.

NFS

ORION-318379: Resolved an issue that could cause sporadic ACL inheritance errors when creating subdirectories through NFS.

NFSv4

ORION-297653: Changed the NFSv4 response code that the VAST cluster returns for a GETATTR request when a user or group is not found in the cluster's internal database. Starting with 5.5.0, the cluster returns BADOWNER. Prior to this change, SERVERFAULT was returned.
ORION-238708: Resolved an issue that could cause the trash folder to enforce a more restricting policy than expected, resulting in occasional permission denied errors for NFSv4.1 clients attempting to move files to the trash folder.
ORION-214943: Improved processing of NFSv4 over RDMA requests to prevent a flow that could cause the requests to fail with the NFS4ERR_DELAY error.

SMB

ORION-349441: Resolved an issue that could cause intermittent STATUS_LOGON_FAILURE errors when establishing an SMB session.
ORION-332901: Enhanced processing of SMB QUERY DIRECTORY requests to prevent potential duplication (in the cluster's response) of file_id values across different files, which could cause I/O errors when a macOS client accesses a directory on a VAST SMB share.
ORION-266324: Resolved an issue that could cause an assertion failed: (base_ctx->get_snap_line().snap_time == P::MAX_SNAP_TIME) alert when handling a scenario that involved placing an SMB lock on a file on a replication source, a failover to the replication target, and attempting to read from the same file.
ORION-186427: Resolved an issue that could sporadically prevent users from logging in via SMB using their Active Directory credentials.
ORION-185514: Improved handling of SMB2 CHANGE_NOTIFY requests to eliminate a flow that could cause a CNode container restart.

S3

ORION-365253: Eliminated a contention that could occur when processing a specific S3 workload with S3 bucket notifications enabled and cause multiple waiting for a lock for too long, IO is stuck - should close connection and timeout expired for @life_type=16,life_gen=<...> (TRAVIS) errors on the CNodes.
ORION-306476: Improved handling of retries for stuck HTTPS sockets to mitigate performance impact on S3 workloads.
ORION-275401: Fine-tuned an internal timeout to prevent a flow that could cause slow response when processing HTTPS S3 requests.

Block

ORION-375653: Resolved an issue that could cause CNode containers to restart during live monitoring of block volumes.
ORION-359297: When listing or viewing block volumes, the VMS includes information about the volume creation time.

Event Publishing

ORION-380946: Resolved an issue that could occur when creating a VAST event topic with a certain configuration and cause the CNode container to restart.
ORION-375322: Added a validation to disallow removing the Kafka protocol from a view that contains Kafka topics.
ORION-375196: Made updates to prevent tenant association-related errors in some scenarios involving deletion and recreation of VAST Event Broker views on multiple tenants.

VAST DataBase

ORION-307379: Resolved an issue where an attempt to provide invalid input for a VAST Database operation resulted in a CNode container restart.
ORION-278671: Eliminated a potential deviation of the size displayed for a projection table in the VAST Database UI (DataBase -> VAST Database -> navigate to the table) from the capacity used by the table according to the Element Store -> Views page.

VAST DataEngine

ORION-333926: Improved caching of local users' keys to ensure that when the cluster admin generates new keys for the user while the user is logged in to the VAST DataEngine UI, the new keys become valid without the need to wait till the VMS key cache entry expires.
ORION-292066: Made updates to avoid an Invalid value error when using VAST DataEngine UI to connect to a Kubernetes cluster from a VAST tenant that has upper-case letters in its name.

Data Protection

ORION-193694: Added support for restoring a Suspended protected path so that the restored protected path points to the original directory. Prior to this change, the restored protected path would point to a temporary directory.

Replication

ORION-327378: Eliminated an intermittent race condition that could cause the CNode container to restart with the assertion failed: (existing_phandle.is_valid()) found an existing link with no valid handle?! error when running asynchronous replication.
ORION-288359: Eliminated a flow that could cause a chain of repeated full synchronization cycles on S3 replication streams.

Authentication and Authorization

ORION-339595: Made updates to ensure that rotation of Active Directory machine account credentials does not cause subsequent failures in Kerberos authentication of SMB access.
ORION-331221: Resolved an issue where attempts to join Active Directory from a cluster where NTLM authentication was disabled for the provider, could not succeed due to The request is not supported errors.
ORION-196963: Resolved an issue where the owner for files and folders created on an NFS4.1 view could be occasionally reported as nobody instead of the correct value. This issue could occur if both LDAP and Active Directory providers (with different domain names) were configured for the view’s tenant and the same group existed on both providers, but the user was part of this group on the LDAP provider only.

VMS

ORION-355322: Resolved an issue that could cause the VMS to get stuck in case it used Supermicro Gen5 as the box type following a hardware discovery flow failure.
ORION-346530: Added an alert to be raised if an NLM lock is in pending queue for a long period of time (by default: 12 hours).
ORION-293054: Made updates to avoid duplicate entries of vast_fan_metrics_hardware_rpm and vast_fan_active metrics for Ceres v2 DNode fans.
ORION-292157: Resolved an issue that could cause the VMS to become unresponsive due to a stuck internal database query.
ORION-285195: Added a validation to ensure that the cluster's PSNT value entered during cluster installation does not contain any illegal characters (such as a trailing whitespace).
ORION-282603: Made updates to preserve user-made changes to the csi role (one of default administrative roles on the cluster) during an upgrade.
ORION-267200: Resolved an issue that could cause a CBox removal process to get stuck with a very large amount of intra-cluster SSH connections open.

VAST Web UI

ORION-355178: Made updates so that the Acknowledge functions in the Alarms and Events page can be used to acknowledge the selected alarms as expected.
ORION-325336: The Analytics -> Analytics page features a new object selection dialog that provides a listing of objects that can be selected for a given analytics category together with detailed information about each object.
ORION-317973: Improved handling of usernames of users from child domains by VAST Web UI so that they are recognized as such and processed accordingly. Querying a child domain user from VAST Web UI no longer returns details for same-named users in the parent domain. This update also enables you to create S3 keys for the correct child domain user in VAST Web UI.
ORION-260502: Updated the logic behind the Capacity Estimation page (Analytics -> Capacity) to avoid including some of VAST Database tables in capacity estimations.
ORION-239505: Improved the logic behind the syslog operation selection toggles in the VAST Web UI (Settings -> Notifications -> Syslog Setup) to ensure that each toggle enables logging of its respective type of operations only.
ORION-175189: Made updates to show the Leading GID and Primary group SID fields as empty when querying a local user using the Aggregated context. Prior to this change, a value of -1 was displayed in this case.

VAST CLI

ORION-156628: Resolved an issue that could cause the 'ViewPolicyProtocolsAudit' object has no attribute 'get' error when attempting to run the viewpolicy show --audit command.
ORION-116206: Added a new filter to the dnode list command, --dbox-name, that lets you display DNodes associated with a particular DBox.

Platform and Control

ORION-350628: Resolved an issue that could cause multiple CNode container restarts with the waiting for lock for too long error followed by temporary service disruption.
ORION-345505: Enhanced the logic for flagging the nodes as failed to prevent a flow where a DNode failure could cause multiple CNode container restarts.
ORION-335190: Resolved an issue that could cause CNode containers to restart on an attempt to calculate Protocol Data Unit (PDU) header digest.
ORION-291390: Introduced updates to avoid service disruption in a scenario where most of the drives become unresponsive but the DNode and the platform are still operable.
ORION-287897: Introduced updates to better balance full vs empty drives across the cluster.
ORION-285271: Resolved an issue where following a VAST OS upgrade, an alert on misconfigured CNode CPU isolation (isolcpu) was raised.
ORION-279766: Resolved an issue where an attempt to remove an Active Directory provider caused the CNode container to restart with an assertion failed: ((true) == (res)) (1 == 0) error.
ORION-266439: Eliminated a potential memory leak that could cause an allocated 90% of mooktze buffers! top consumer is WB_STRIPE_POOL_MIGRATING_USAGE_RECORD alert on the cluster.
ORION-247833: Improved management of platform metrics collection routines to avoid flows that could result in metrics collection being stuck.
ORION-215810: Resolved an issue that could prevent the cluster from electing the leading node after a power outage.

Callhome and Support

ORION-255750: Made updates to automatically remove the <bundle>.tar.progress file when the bundle upload is finished.

Limitations

Install and Upgrade

Automatic drive firmware upgrade is not performed on drives that have been moved manually from an old DBox to a new DBox during the DBox replacement procedure. (Note that this does not apply to EBoxes.)
VAST DataEngine requires a special upgrade procedure performed in strict compliance with the steps and guidelines provided in the VAST Cluster Administrator's Guide.
ORION-281679: SSD firmware upgrade is not supported for Supermicro EBoxes.
ORION-280966: Having imported a JSON configuration file in VAST Cluster Install, you need to manually verify that all the populated fields have expected values. Sometimes, depending on the cluster configuration and environment, some of the fields are not populated as expected during the import.
ORION-242658: BMC firmware upgrades are not supported for Supermicro Genoa CNodes.
ORION-222648: NDU that includes automatic adjustment of CNode CPU isolation settings (isolcpus) is not supported for EBoxes.
ORION-214559: A BMC upgrade cannot be performed with an inactive CNode that has been powered off.

Networking

The following limitations apply when using Open Telemetry-Based Ethernet Network Monitoring:
- Up to 70 switches per cluster.
- The switch must have OTel configured to send telemetry data over gRPC to the VMS IP address using port 4317.
- TLS on the connection used to obtain the metrics from the switch is not supported.
- Supported on NVIDIA Cumulus switches 5.12.1 and later. For other switch vendors, inquire VAST Support.
The following limitations apply when implementing L3 networking:
- After enabling L3 access for a virtual IP pool, it cannot be disabled.
- L3 access is not supported on virtual IP pools for which CNode Port Affinity is configured.
- ORION-333743: L3 access is not supported on virtual IP pools that include more than one range of virtual IPs.
- L3 networking is not supported on IB clusters.
- L3 networking is not available for VAST on Cloud.
- MD5 authentication is not supported.
- ORION-266297: When using numbered BGP, each CNode can only be involved in one BGP configuration. To learn which CNodes are currently used in which BGP configurations, use the Network Access -> Virtual IP Pools page that displays CNodes and BGP Configurations for each virtual IP pool.
- Reverting an L3-configured cluster to L2 through VMS is not supported. If you need to revert, contact VAST Support.
Changing network configuration from VMS (in VAST Web UI: Settings -> Configure Network) is not supported for clusters that have any CNodes that are connected simultaneously to multiple separate client data networks.
ORION-380243: When reconfiguring cluster networking from non-B2B to B2B, there is no indication of the requirement to physically swap the cables on the nodes so that the task can complete successfully. The task pauses when attempting to deactivate the final CNode and resumes only after the cables are reconnected.
ORION-374025: Virtual IP pools with a role of Query Engine cannot be modified after creation. This is a built-in safeguard to prevent in-flight queries from failing. If you need to introduce a change into an existing Query Engine pool configuration, delete the pool and create a new one.
ORION-340819: Before modifying a virtual IP pool, ensure that during the modification all of the views associated with this pool remain assigned to the pool. Switching a view from one pool to another while either old or new pool is being modified, may result in unexpected behavior.

Encryption

The following rules and limitations apply when using Kerberos providers:
- MIT Kerberos providers are used for NFSv4 only.
- A tenant can be associated with a single Kerberos provider only.
- Up to eight Kerberos providers can be defined on a VAST cluster.
- Kerberos principals used to authenticate to the VAST cluster must exists as users in an LDAP provider or in a VAST provider associated with the same tenant as the Kerberos provider. Otherwise, they are squashed to the nfsnobody user.
The default tenant does not use keys created or stored within an EKM. To encrypt data within the default tenant with EKM-based keys, use encrypted paths.
ORION-388667: Creating an encrypted path on a directory exposed by an ABAC-enabled view is not supported. An attempt to create an encrypted path on the same directory as the one exposed by an ABAC-tagged view causes the assigned ABAC tags to be removed.
ORION-208004: Enabling VAST OS boot drive encryption requires that the node is inactive. Enabling the encryption on an active node may cause a long reboot sequence.

Quotas

The following limitations apply when using Remote Quota Protocol (rquota):
- SETQUOTA requests are not supported.
- Quota information can be provided for active users only.
- Information about group quotas is not provided.
Quotas are not enforced on replication destination directories under a protected path. For example, if the protected path is /ppath, a quota on /ppath/yourdir is not enforced.
ORION-301453: Reaching a quota's hard limit blocks writes to existing (previously written) offsets. When using ESX, this results in inability to perform operations such as VM clone, VM power-on, or datastore operations.

Lifecycle Rules

ORION-280461: The maximum size of a lifecycle rule is 15MB.

Quality of Service (QoS)

The prioritization flag cannot be set for user QoS policies.
Some high-priority optimizations are applied to NFSv3 only.
The following limitations apply when using QoS for block volumes:
- Using more than 100 volumes that have QoS limits applied to them may result in inaccurate performance capping.
- Prioritization of a volume QoS policy over cluster QoS limits is not supported.
- ORION-266523: For volumes that share the same subsystem, it is recommended that all of the subsystem volumes have the same QoS limits set, or all of them do not have a QoS policy assigned, to prevent possible host IO performance impact.

NFS

ORION-324346: NFS aliases are not supported with VAST implementation of Remote Quota Protocol (rquota).
ORION-115336: If one creates an NFSv4.1-only view and mounts it, and then creates its parent view with NFSv3 only, IO operations on the NFSv4.1-only view succeed but mounts are not allowed.

NFSv4

TLS encryption with NFSv4.1 is not supported for NFSv.4.1 over RDMA.
The following requirements and limitations apply when using NFSv4 file delegations:
- The NFSv4 protocol requires Network Time Protocol (NTP) to be configured on both the VAST cluster and the client. While VAST does not enforce this requirement, the absence of proper time synchronization may, in rare cases, result in unexpected behavior.
- The client must have an NFSv4 backchannel connection.
- For multi-protocol views with NFSv4 delegations enabled, close-to-open consistency is only preserved between NFSv4 clients, but not between NFSv4 clients and other protocols (including NFSv3). For example, if a non-NFSv4 client modifies a file on such a view, the modification may not become immediately visible to an NFSv4 client holding an active delegation on the file.
- Before making a path read-only by using the /folders/read_only endpoint of VAST REST API, ensure that NFSv4 delegations are disabled or returned. Otherwise, unexpected behavior may be encountered.
- Restoring a snapshot on a tenant that have NFSv4 delegations enabled, may cause unexpected behavior with regard to delegated files. It is recommended to revoke existing delegations before restoring a snapshot.
- VAST Cluster recalls granted NFSv4 delegations when a failover process is started.
- In some cases when there is very high NFSv4 callback load per cluster or per client, VAST Cluster might start recalling or revoking NFSv4 delegations due to callback request timeouts.
- NFSv4 delegations require mounting the client at least as NFS version 4.1. It is recommended to mount as NFS version 4.2 for additional performance improvements. Linux kernels 6.11+ on a client provide more performance advantages (RFC9754), which get backported to older kernels through VAST-NFS.
- If you are using VAST-NFS, note that NFSv4 delegations are supported with VAST-NFS 4.0.32 and later. VAST-NFS 4.5 includes enhancements related to support of NFSv4 delegations.
- SUSE Linux Enterprise Server 12.x and CentOS 7.x clients are not supported with NFSv4 delegations enabled.
The following applies when using NFSv4 extended attributes:
- The VAST cluster does not enforce any limitation on the maximum number of extended attributes per file. However, some clients may have such a limitation when listing extended attributes.

SMB

The following limitations apply when using SMB hardlinks:
- Creating a hardlink to another SMB-enabled view is not allowed.
- Creating hardlinks to directories is not allowed.
- Linking an Alternate Data Stream (ADS) is not supported.
- Creating a hardlink on a Global Access satellite cluster is not supported.
- Creating a hardlink, on the Global Access origin cluster, of a file that has already been opened from a satellite cluster, is not allowed.
The following limitations apply when managing SMB open files:
- In VAST Global Namespace, operations on open file handles are only supported for the current cluster.
- ORION-289428: SMB open file queries do not show open SMB hardlink destination files.
- ORION-288057: When listing files on a path where there is an open ADS stream (path\file:stream), the stream is shown as if the file is a folder (\path\file\stream).
- For a path with an open Alternate Data Stream (ADS) on it, the following actions are not supported:
  - Closing the ADS
  - Listing individual open handles associated with the ADS.
Views exposed as SMB shares work only if the cluster is joined to Active Directory. This includes both SMB-only and multiprotocol views.
The following limitations apply when using SMB change notifications for subdirectories:
- SMB recursive change notifications are not available for paths for which Global Namespace is configured.
- Using SMB recursive change notifications for nested SMB shares may entail unexpected behavior.
The following limitations apply when using Alternate Data Streams (ADS):
- ADS are not seen or accessible by protocols other than SMB.
- ADS will not be retained when files and directories are replicated to S3.
  ORION-169707: When the Hyper-V management tool tries to list VAST Hyper-V SMB shares on an SMB server, the The RPC server is not available error can occur if the SMB server is specified using its FQDN. To avoid this error, specify the IP address of the SMB server instead of the FQDN.
ORION-160323: After updating permissions for an SMB share in Windows Explorer, a duplicate SMB share can be displayed. The duplicate SMB share disappears upon a refresh (F5).
ORION-134730: An attempt to restore a file can fail if after the restore has started, a quota is set on the path where the file resides.

S3

The following limitations apply when using S3 Versioning:
- After versioning has been enabled on a bucket, it cannot be disabled.
- Multiprotocol access to versioned objects is not supported. NFS and SMB clients may attempt to access the S3 versioned objects in read-only mode.
- MFA Delete is not supported.
- S3 conditional writes are not supported for versioned buckets.
The following rules and limitations apply when using S3 Server-Side Encryption with Customer-Provided Keys (SSE-C):
- If an S3 request contains an x-amz-server-side-encryption-customer-* header, it is required to use HTTPS for the request to be accepted by the cluster.
- Only AES256 encryption algorithm is supported.
- Objects encrypted with SSE-C cannot be accessed through other access protocols.
- For replication environments, it is strongly recommended to avoid using SSE-C unless all of the clusters involved support SSE-C.
The following limitations apply when using S3 presigned POST requests:
- An object to be uploaded via a S3 presigned POST request must have only ASCII characters in its name.
- A POST policy used for S3 presigned POST requests can be up to 4800 bytes.
The following limitations apply when using S3 Indestructible Object Mode:
- An S3 Bucket view with Indestructible Object Mode cannot have other protocols enabled.
- Indestructible Object Mode cannot be set for a view that points to / (root directory).
- It is not allowed to have views under the view in Indestructible Object Mode, or at the same path as the Indestructible Object Mode view.
- Directories that contain views that have indestructible object mode enabled cannot be moved to the trash folder.
- Indestructible Object Mode cannot be used together with S3 Object Locking or S3 Object Versioning.
- Indestructible Object Mode cannot be used together with WORM.
- Indestructible Object Mode cannot be set for a view that exposes the protocol audit log directory.
- A snapshot cannot be restored on a view that is in Indestructible Object Mode.
- Views in Indestructible Object Mode are not subject to replication or Global Access.
The following limitations apply when using CORS:
- ORION-366116: VAST implementation of CORS does not include unique IDs for CORS rules.
- ORION-314829: VAST Cluster does not include the Access-Contol-Allow-Origin header in case of a failed PutObject request.s
ORION-340559: Absolute-form URLs (e.g. http://.../.../<object> instead of /<object>) in CompleteMultipartUpload requests are not supported.
ORION-240888: S3 bucket monitoring does not take into account VMS-originated S3 requests, including those related to features such as:
- Lifecycle rules
- Event notifications
- Bucket policies
- Object ownership
- Bucket versioning
- Object locking
- Bucket logging
ORION-197281: VAST Cluster disables bucket logging set on a bucket from which data is synchronously replicated to another bucket once you set up bucket logging on the replication destination bucket and configure it to use a different logging destination bucket.
ORION-190674: Once created, an S3 bucket cannot be renamed or moved to a different path. Thus, for example, if you try to change the bucket’s path when modifying a view in VAST Web UI, the change does not take effect and the view will still be listed with the old path.

Block

VAST Cluster does not support subsystem names that contain a space (in VAST Web UI: Element Store -> Views -> choose to create a view -> Block tab -> Subsystem name field).
Snapshots on local protected paths are allowed but replication on non-local protected paths is not supported.
For Rocky Linux-based clients, VAST recommends that the client uses Rocky Linux 9.4 or later.
An attempt to remove a volume that has snapshots may cause errors for volume objects of snapshots of that volume, if they exist.
A view that is used to expose block storage cannot have other storage protocols enabled.
The following VAST capabilities are not available with block views:
- Access control features (such as ABE, ABAC, WORM)
- VAST Audit Log
- Replication to a remote peer
- Global Access
- Remote global snapshot clones
Nesting of a block view inside an existing block view is not allowed.
The host NQN cannot be modified. To change the NQN, you need to remove the host and then add and map it anew.
You cannot enable or disable block storage support on an existing view. Block storage support can only be enabled for a view during view creation and cannot be disabled afterwards.
Block devices can be created on empty directories only.
If a host defined on the VAST cluster does not have any volumes mapped to it, NVMe auto-discovery does not show this subsystem.
When using the VAST Web UI or CLI options to bulk create volumes or hosts, the number of items to be created cannot exceed 256. When mapping hosts to volumes, up to 256 items can be mapped at a time.
The maximum IO block size is limited to 1MB (4GB for unmap).
ORION-388015: Compare-and-Write (CAW) operations are only supported when performed against a single block (512B).

Attribute-Based Access Control (ABAC)

The following features and capabilities cannot be used together with ABAC-tagged views:
- If a tenant has ABAC-tagged views, you cannot change or remove the Active Directory provider configured for the tenant.
- When using NFSv4, it is not allowed to create hardlinks in views that have ABAC tags.
- When using S3:
  - ABAC cannot be used with anonymous S3 access. You cannot set ABAC tags for views that have anonymous S3 access enabled.
  - It is not allowed to set ABAC tags on a view that is a target for S3 bucket logging.
  - Requests from S3 superusers are handled in the same way as for regular users. This means that an S3 superuser is not granted access if the ABAC access check denies access for this user.
- A directory under which an ABAC-tagged view exists, cannot be moved to the Trash folder.
- Bulk permission updates are not available for ABAC-tagged views.
- Lifecycle rules cannot be set for files or directories with ABAC tags.
- The tenant with ABAC-tagged views cannot have SMB native authentication enabled.
ABAC is supported on views controlled with SMB, S3 Native and Mixed Last Wins security flavors. ABAC is not supported with NFS flavor.
ABAC tags cannot be set on the cluster’s root directory (/).
If a user does not have any ABAC permissions, the user still can mount an NFSv4 export or map a SMB share to a local drive, but the user is not allowed to perform any operations on the files or directories.
Once assigned, you cannot edit or remove the ABAC tags of a view. Assigning new ABAC tags to an existing view or directory (storage path) is not allowed.
ABAC is not supported with NFSv3.
If you create a view for a directory that already exists, ABAC tags from the existing directory are assigned to the newly created view. In this case, there can be a delay between the view creation time and the time when the view's ABAC tags can be displayed.
After a child view inherits ABAC tags from the parent view, you cannot update or remove the ABAC tags on the child view.
ORION-388667: Creating an encrypted path on a directory exposed by an ABAC-enabled view is not supported. An attempt to create an encrypted path on the same directory as the one exposed by an ABAC-tagged view causes the assigned ABAC tags to be removed.

Write Once Read Many (WORM)

You can add additional protocols to a WORM-enabled view, but cannot remove a protocol.
You cannot configure a WORM-enabled view for NFS/SMB and S3 together.
Nesting views under a WORM-enabled view is not allowed.
You cannot disable WORM for the view once it is enabled.
Bulk permission updates are not allowed on WORM-enabled views.

User Impersonation

ORION-216379: When VAST protocol auditing is enabled on a user-impersonated view, only UID of the original user is included in the log. The user's login name and SID are not included.

Bulk Permission Updates

A bulk permission update can run only when the target view (the view exposing the files and directories for which you want to update permissions) is on the same tenant as the template view.
Only one bulk permission update task per tenant can run at a time.
Running a bulk permission update on a view where the security flavor does not match that of the template view may result in inaccessible or incompatible permissions set.
Read-only snapshots and VAST special directories (.vast in S3 buckets, .trash, .snap, .remote) are excluded from bulk permission update.
If a client attempts to set permissions on directories or files being updated via a bulk permission update, the result is unpredictable.
Bulk permission update cannot run on ABAC-tagged views.

VAST Audit Log

ORION-385315: When accessing VAST Audit Log through the VMS REST API (including VAST Web UI), sorting/ordering of VAST Audit Log entries is not supported in case the query results include more than 1000 rows. In this case, VAST Audit Log returns random 1000 entries. To mitigate this behavior, narrow your search to return less than 1000 entries, or use other access mechanisms (such as VAST DB SDK or a query engine connected to the VAST Database).

Event Publishing

The following limitations apply when using VAST Event Broker:
- Producer API:
  - Messages are limited to 1MB.
  - In the event record, the key is limited to 126KB and the value is limited to 126KB.
  - Access to topics by UUID is not supported.
  - Idempotent producing is not supported.
  - Automatic creation of topics is not supported.
- Consumer API:
  - No more than 256 consumer groups per view (broker)
  - The following is not supported:
    - Consumer group stickiness parameters (such as group.instance.id)
    - READ UNCOMMITTED isolation level
    - Cooperative rebalancing
    - Client rack awareness
    - Fetch sessions (only full fetch will be applied), delayed fetch parameters
    - Seek by time
- Admin API:
  - Supported APIs include the APIs to create topics, delete topics, and to delete groups, as well as describeConfigs and alterConfigs APIs that let you get and update the topic configurations.
- The following Kafka capabilities are not supported:
  - Over-the-wire compression of messages
  Note: VAST compression of data is supported.
  - Transactions
- Only one virtual IP pool can be associated with a Kafka-enabled view, providing at least one virtual IP per CNode. Once the view has been created, the virtual IP pool cannot be replaced by another one (but it can be modified if needed).
- The amount of VAST Event Broker views that you can create on a VAST cluster, is limited by the maximum number of views supported by the cluster and by the maximum number of virtual IP pools (since each broker view requires a dedicated virtual IP pool). See VAST Cluster Scale Guidelines for details.
- The amount of event topics that you can create on a VAST cluster, is limited by the maximum number of tables per VAST Database table and the overall amount of event topic partitions.
- A topic can have up to 20,000 partitions. The number of partitions in a topic cannot be changed after the topic has been created. Up to 200,000 partitions are supported per VAST Event Broker view.
- Event queries based on the topic partition are not supported.
- When listing consumer groups, the response is limited to 256 groups per Kafka-enabled view.
- Event publishing and consuming operations, as well as topic management operations are not subject to VAST Protocol Auditing or Quality of Service (QoS).
- VAST Cluster supports Confluent Kafka Python client 2.4 - 2.8. The aiokafka client is not supported.
- Authentication and authorization:
  - Active Directory/LDAP is not supported. The user must be defined as a VAST local user.
  - Only one Kafka TLS certificate can be uploaded per VAST cluster.
- Data protection:
  - The Kafka-enabled view needs to be manually created and associated with a virtual IP pool at the destination peer. The pool must have the same name as the one at the source peer.
  - Fast restore of a protected path containing a Kafka-enabled view is not allowed.
  - VAST replication of consumer groups is not supported. Consumer group offsets are not replicated.

VAST DataBase

The following limitations apply when using vector indexing:
- Once vector indexing is enabled for a table, it cannot be disabled.
- After a vector index is created, its distance function cannot be changed.
- Only one vector column can be indexed per table.
- The word 'vector' cannot be used as the vector column name.
- Adding a vector index to an existing table is not supported.
- Adding or deleting columns in an indexed table is not supported.
- Vector indexing cannot be used together with sorted tables and partitioned tables.
- VAST Connectors for Trino, Spark and Dremio do not support vector index-based search.
The following limitations apply when using table views:
- View properties are not supported.
- Queries to a view must include full table names.
- Redefining a view is supported for Spark clients only.
- User-defined column names and comments are lost if the schema of the query changes when redefining a view.
The following limitations apply when using table partitions:
- Partitions cannot be created based on nested columns (list, map, struct column types).
- Columns used to create partitions can only be updated if the update does not affect existing partitioning.
- Columns used to create partitions cannot be dropped once the partition exists.
- Table partitions cannot be used together with the following database features:
  - Vector indexing
  - Semi-sorted projections
  - User-defined row IDs
- Sorting can be used on partitioned tables, but the sorting order can only be defined at table creation. Existing partitioned tables cannot be sorted.
- It is not allowed to use the same column both for sorting and partitioning.
- ORION-302884: Quotas on partitioned tables are not supported.
- Replication of partitioned tables is not supported.
The following limitations apply to sorted tables:
- Once enabled for a table, the sorting cannot be disabled.
- Sorting can be performed for tables that contain 512k or more rows. Although you can enable sorting on a table regardless of its size, smaller tables do not get sorted.
- Once sorted, tables cannot be unsorted or have their sorted columns changed.
- Only one sorting order can be defined on a table. Up to four sorted columns are supported.
- Sorted column values cannot be updated to different values.
- Sorted tables cannot be replicated. Tables that are replicated cannot be sorted.
- Sorting cannot be enabled on tables that have semi-sorted projections. Semi-sorted projections cannot be added to sorted tables.
- Nested data types are not supported for sorted columns.
- Tables that expose the internal row ID with the vastdb_rowid column cannot be sorted.
The VAST connector for Trino does not support the SECURITY clause in CREATE VIEW statements.
The following requirements and limitations apply when using Row/Column-Level Security:
- Row filtering and column masking are supported for Trino query engines only.
- The Trino user's identity policy must have the EndUserImpersonation and GetRowColumnSecurity actions allowed.
VAST Query Engine does not support the nested data type of map.
A VAST Database view (a view with the Database protocol) can only be created on clusters with data reduction enabled.
The following limitations apply when using Blob Expansion:
- Only one expansion table can be defined for a given column in the event topic.
- The expansion target table must be created (without columns) before you start configuring blob expansion for the table being expanded.
- Renaming or removing target table columns that are included in blob expansion configuration, is not allowed.
- Updates/deletes to the source columns do not affect the target table.
- Renaming of the source table is not allowed.
ORION-353292: Importing data into a VAST Database table for which sorting and partitioning have been enabled, takes considerably longer compared to import to an unsorted and unpartitioned table.

VAST Catalog

The maximum path length supported by VAST Catalog is 1024 characters.
ORION-197741: VAST Catalog cannot be enabled on a cluster that uses encryption keys managed through EKM, including per-tenant and per-path encryption keys.

VAST DataEngine

The following rules and limitations apply when running Trino clusters on VAST:
- Not less than three CNodes are required per Trino cluster (one for the coordinator, two or more for the workers).
- In case of CNode HA events, the Trino cluster remains accessible and running queries as long as one coordinator CNode and one of the worker CNodes are up.
- A VAST cluster upgrade (NDU) performed while running a Trino cluster may cause Trino query failures. The Trino cluster will be fully operational after the NDU is complete.
If VAST DataEngine is enabled on the cluster, replication of the tenant's root directory is not allowed.
The following requirements and limitations apply when using NFS-based triggers:
- Only regular files can be the source for trigger events. Directories, symlinks and other special file types are not supported.
- Only NFSv4 is supported. Events resulting from NFSv3 or SMB operations on the same source view would not be used for the trigger.
- The source view must have NFSv4 and S3 Bucket protocols enabled.
- NFSv4 file delegations must be disabled on the view configured as the trigger's event source view.
- Events occurring to elements on nested views of the source view, cannot be used as the basis for VAST DataEngine triggers. For example, if an NFS-based trigger is configured on a source view at /abc/, an event on its nested view /abc/xyz/ will not trigger any action in VAST DataEngine.
- If a file is renamed or deleted while it is being created (e.g. the file has been opened and written to, and then it got unlinked or removed), no create or delete event is produced.
- ORION-320111: During an upgrade, the VAST cluster does not produce Element Deleted events for files subject to NFSv4 unlink and rename operations.
Up to 32 engines are supported per VAST cluster.
The following rules and limitations apply when using VAST compute clusters:
- Each compute cluster requires at least three CNodes (because three server nodes are required to support HA). If more than 15 CNodes have been selected for a compute cluster, five master nodes will be allocated.
  
  Note: The role of the node (server/worker) is determined automatically considering the lifecycle state of the nodes, and may change throughout the lifecycle of the VAST cluster (for example, role changes are expected in maintenance operations such as FRU and NDU).
  
  Depending on the number of active worker nodes, server nodes may also execute workload.
- A CNode can be used for one compute cluster only.
- ORION-348513: Compute clusters cannot be deployed in a environment with CNode port affinity (where the LEFT port and the RIGHT port of a CNode can be on different networks).
- Up to 32 VAST-internal compute clusters can be created per VAST cluster.
- ORION-291582: VAST-managed compute clusters are not available in VAST on Cloud.
- ORION-365247: VAST-managed compute clusters cannot be run on CNodes with L3 networking.
- ORION-288369: VAST-managed compute clusters are not supported in environments with Layer 3 data networking.
- ORION-371504: IPv6 addressing is not supported.
- ORION-342312: There is no validation that the IP specified for a compute cluster node is free.
- ORION-397536: VAST-managed compute clusters cannot be run on ENodes.

Data Protection

When using NFSv3, in rare cases with large numbers of files and directories, the existence of a view with Global Synchronization enabled under a protected path can block the removal of the protected path.

Replication

The following limitation applies to VAST Database asynchronous replication:
- ORION-179909: VAST Database asynchronous replication cannot be used together with Global Access or synchronous replication on the same path.
- Only committed database transactions are replicated.
- VAST Catalog and audit log are not replicated.
- Multi-tenant replication of VAST Databases is not supported.
The following limitations apply to synchronous replication:
- A protected path using synchronous replication can only contain S3 buckets. You cannot use synchronous replication for paths that are exposed to any other protocols.
The following limitations apply to synchronous replication for S3:
- Synchronous replication is supported for S3 buckets only.
- It is not allowed to configure local snapshots, global snapshot clones or Global Access on the protected path for which synchronous replication is configured.
- S3 lifecycle rules are not replicated.
- S3 keys are replicated asynchronously.
- Synchronously replicated directories are not subject to bulk permission updates.
Protected paths with asynchronous replication cannot be nested.
Data cannot be moved into or out of a path that is protected by either asynchronous replication or S3 replication.
it is not possible to change which protection policy controls any replication stream.
The following limitations apply to bucket replication:
- ORION-338118: If your replication environment includes a cluster where bucket replication is enabled, ensure that all of the replication peers also have it enabled. Otherwise, unexpected behavior may be encountered during asynchronous replication failover after the clusters are upgraded to version 5.5.
The following block-specific rules and limitations apply when using asynchronous replication of block volumes:
- If a replication group includes a replication stream that replicates block volumes:
  - All replication peers in the group must be running VAST Cluster 5.5.0 or later.
  - The replication group can include only asynchronous replication streams.
- Block hosts and mappings between the volumes and the hosts are not replicated.
- The default block view policy is to be created manually for the relevant tenant at the replication destination peer.
- When using asynchronous replication of block volumes with ESXi, user intervention is required to unmap volumes from the ESXi host if they are no longer writeable (e.g. after a failover). Otherwise, volumes that are intended to be presented as read-only, are not being detected as such by ESXi.
ORION-208123: Local user accounts are not subject to replication.

Global Access

VAST Database and VAST Event Broker (Kafka) views are not supported.
The following limitations apply when using Global Access for S3 buckets:
- Identity policies must be enabled at the cluster to which they get replicated.
- The following VAST capabilities are not supported on destination buckets:
  - S3 event notifications
  - S3 Indestructible Object Mode
  - Lifecycle policies
  - Write Once Ready Many (WORM)
- Bucket logging is only supported if both the source and destination buckets are in the same protected path.
- Bucket replication between two clusters is only supported when the bucket is associated with the default S3 view policy.
- S3 endpoints are not replicated.
The following combinations of access protocols and security flavors are supported:
- Access to a Global Access protected path (global folder) on the source cluster: NFSv3, SMB, S3 with any of the flavors; NFSv4 with any flavor except the SMB flavor.
- Access to a satellite path on the destination cluster: NFSv3 with NFS or S3 Native flavor, SMB (all flavors), S3 (all flavors)
- Access to a path which is part of both Global Access and asynchronous replication setup: NFSv3 with NFS flavor, SMB (all flavors), S3 (all flavors)
  If a view is configured with both NFSv4 and SMB, it must be controlled with the NFS security flavor.
Lease expiration time can only be set when creating a global access protected path. You cannot change lease expiration time when you modify a global access path.
VAST Catalog does not provide information on the cached data on the remote cluster.
ORION-194805: Applications that use SMB2 Byte Range Locks are not supported when the SMB client is connected via a remote Global Access protected path. Examples of such applications are Microsoft Office suite on macOS, Microsoft Hyper-V, AutoDesk 3ds Max and some Adobe Premiere plugins.
ORION-194613: If some files have additional hardlinks, the amount of bytes reported as prefetched can be higher than the actual amount prefetched.

VAST DataSpace

VAST DataSpace requires that each cluster participating in the inter-connection is running VAST Cluster 5.0 or later.
VAST clusters with IPv6 management IPs cannot be added to VAST DataSpace. Only IPv4 is supported.
ORION-135966: The inter-connecting clusters must have connectivity to each other through the clusters' management networks.

VAST on Cloud (VoC)

VAST Cloud (Polaris) UI does not provide an option to run a cluster upgrade. Cluster upgrades are to be performed using the VAST Web UI.
VAST on Cloud clusters do not support OS upgrade.
In the event of downtime, data is rebuilt while the cluster comes back online. In case of a subsequent failure during the rebuild, data integrity is not guaranteed.
ORION-327048: For VoC on GCP, it is recommended to have a multiple of 8 non-VMS VMs per cluster, since GCP has 8 failure domains and imbalance may result in failed stripe allocations.
ORION-145141: Creating a tenant with EKM encryption is not supported on VoC clusters.
ORION-113036: After you reregister the same VoC cluster in Uplink, information about the previously registered instance of this cluster is no longer available in Uplink.

Authentication and Authorization

The following rules and limitations apply when using STS, IAM roles and OIDC providers:
- STS is supported with HTTPS only.
- Use of STS requires that the S3 protocol is enabled for the cluster.
- One OIDC provider per VAST cluster tenant.
- When replicating IAM roles:
  - ORION-278747: IAM role replication may take several minutes to complete in case the cluster is under high load or there is a very large amount of roles to replicate.
  - STS access keys are replicated only when the protection path is in SYNC state.
  - ORION-278217: When processing the assume role operation, the cluster may be raising a Failed to replicate STS temp keys alert until the synchronous replication stream/protection path reaches the SYNC state.
- ORION-287563: Only five OIDC keys can be returned for a manual request to get or refresh OIDC keys.
The following limitations apply when using netgroups:
- Hosts should have both forward and reverse DNS entries. When VAST Cluster gets the netgroup hostname response from a NIS or LDAP server, it resolves the hostname via DNS.
- Netgroups are only used to allow or deny clients' access via NFS. VAST Cluster does not accept netgroup entries in host-based access rules for other access protocols.
The following limitations apply to Multi-Forest Authentication:
- VAST Cluster does not allow adding two different Active Directory configuration records with the same domain name but different settings for multi-forest authentication and/or auto-discovery.
- Names of users' domains are not displayed in data flow analytics.
- If a trusted domain becomes unavailable and then recovers, SMB clients can use it to connect to the VAST cluster only after a period of time, but not immediately upon domain recovery.
- Clients cannot establish SMB sessions immediately after a trusted domain recovers from a domain failure.
- If a group exists on an Active Directory domain in a trusted forest and the group scope is defined as DomainLocal, VAST Cluster does not retrieve such a group when querying Active Directory, so members of such a group are denied access despite any share-level ACLs that can rule otherwise.
- If TLS is enabled, the SSL certificate has to be a CA-signed certificate that is valid for all of the domain controllers in all trusted forests. If the certificate is not valid for a domain controller, this domain controller is not recognized.
- ORION-156168: In a multi-forest environment, after migrating a group account from the forest of the cluster’s joined domain to another forest, information about historical group membership is not kept, so users in the migrated group might not be able to access resources to which they used to have access prior to the migration.
The following requirements and limitations apply when using mTLS authentication for NFS:
- mTLS authentication for NFS requires TLS 1.3.
- The VASTNFS driver is required to use mTLS certificate-based tenant identification for NFSv3. (NFSv4 does not require VASTNFS.)
- When using mTLS certificate-based tenant identification for NFSv3, the client must mount with mountproto=tcp.
- Revocation of an mTLS certificate by using CRL Distribution Points (CDP) is not supported.
- When using NFSv3, if a parent view does not have TLS or mTLS enforced, the VAST cluster allows unencrypted access to any of its child views, regardless of whether the child view has TLS or mTLS enforcement configured on it.
- For clients using multiple certificates for different NFS exports (certificate per mount), Linux kernel 6.17 or later is required.
- ORION-332649: A CRL file can contain up to 100 certificates to be revoked.
The following limitations apply when using Kerberos/NTLM authentication:
- ORION-143944: When using Kerberos/NTLM Authentication to authorize SMB users from non-trusting domains, the DOMAIN\username format cannot be used to specify users of remote domains. The username@domain format must be used instead.
- ORION-141763: Before enabling or disabling NTLM authentication, you need to leave the cluster's joined Active Directory domain. After NTLM authentication is enabled or disabled, rejoin the domain.
- ORION-134299: When the tenant is set to use Kerberos/NTLM authentication to authorize SMB users from non-trusting domains, both NFS and SMB must use the native SMB authentication (Kerberos), and not Unix-style UID/GIDs.
The following limitations apply when using an allow-list of Active Directory DCs and GCs for LDAP queries:
- The list must include at least one Global Catalog.
- The list can contain up to 10 entries.
- The Active Directory provider must have multi-forest authentication disabled.
ORION-344017: When using NFSv3, if a parent view does not have TLS or mTLS enforced, the VAST cluster allows unencrypted access to any of its child views, regardless of whether the child view has TLS or mTLS enforcement configured on it.
ORION-202335: If the cluster has Active Directory domain auto-discovery enabled, the discovered domains are kept in cache for quite a long time. If you modify an existing provider's configuration while auto-discovery is on, VMS may still report the old cached entries. To avoid this, rerun auto-discovery or remove and re-add the provider.
ORION-195524: Following a cluster recovery and while the Active Directory provider is still inaccessible, VAST Cluster can resume IO of provider users if they use NFSv3 or NFSv.4.1 with NTLM authentication. IO of provider users accessing through SMB or NFSv4.1 with Kerberos authentication is not resumed during this period.
ORION-187136: Identity policies are replicated as disabled to the destination peer, where if needed, they can be enabled manually.
ORION-152475: An access denied error is returned for NFSv3 or NFSv4 requests if they are checked against an identity or bucket policy with an s3:ExistingObjectTag condition statement in it.

VMS

Tenant client metrics can only be collected for NFSv3 and NFSv4.
The following limitations apply when installing a SSL certificate for VMS:
- Only RSA-generated public keys are supported.
- Password-protected private keys are not supported.
When setting a VMS login banner text via the VAST CLI, multiple lines are not supported.
ORION-212118: If a wrong VMS token is passed, the cluster responds with 403 FORBIDDEN but not with 401 UNAUTHORIZED.
ORION-187584: An empty realm (which does not contain any objects) cannot be assigned to a role.
ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.

VAST Web UI

The visual identity policy builder in VAST Web UI (User Management -> Identity Policies -> choose to create or edit an identity policy) does not include predefined action statements for Kafka-related objects.
ORION-249325: When you create the first block view on the cluster via VAST CLI or VAST REST API, the VAST Web UI does not display the Element Store -> Volumes or Hosts tabs until you refresh the page.

VAST REST API

When listing views through the VMS API, the resulting list is truncated to the first 16,000 entries. To obtain a complete listing, use multiple requests with pagination.

Platform and Control

The following limitations apply to conversion to write buffer RAID:
- Conversion from VAST releases prior to 3.4 is not supported.
- This capability is not supported for clusters with TLC drives, and also for VAST on Cloud clusters.
- The cluster must include the following minimum number of DBoxes:
  
  DBox Type DBox HA enabled DBox HA disabled
  
  Ceres 15 4
  
  Mavericks 22 4
The following rules and limitations apply when using Rack-Level Resiliency:
- Every DBox must be associated with a single failure domain.
- At least seven failure domains must be defined.
- The number of DBoxes in the rack cannot exceed the number in any other rack by more than one.
- The total available DBox capacity in a rack cannot be more than twice the available capacity in any other rack.
- If two of the racks in a rack level-resilient cluster go down, bringing up only one of them does not return the cluster to normal operation. The cluster restores service only after the second of the racks goes up.
The following limitations apply to metadata rewrite (required during upgrade to VAST Cluster 5.5.x from 5.4.x):
- A metadata rewrite process cannot be run while there is another active rewrite, upgrade, expansion, migration, replacement or removal process running. An upgrade to a major version would be blocked if attempted during metadata rewrite.
- Metadata rewrite can only be started if the cluster's handle usage is less than 70% and the metadata block usage is less than 80% (these metrics are shown in the Cluster MD Usage analytics report).
Use of flash write buffers with DBox HA capability enabled requires at least 10 boxes.
The following limitations apply to EBoxes:
- ORION-193794: Power cycling of an EBox where the leader was running may result in significant IOPS degradation until the EBox is up again. Contact VAST Support for a workaround.
- DBox migration is not available for EBoxes.
The following rules and limitations apply when using DBox removal:
- A DBox removal process can only be started after metadata rewrite is complete (metadata rewrite is required when upgrading to VAST Cluster 5.5). See Limitations for metadata rewrite.
- DBox removals reducing the cluster size below the initial installation size may not be allowed.
- Once the DBox removal process has started, it cannot be stopped.
- Only one DBox removal process can be run at a time. The process can involve one or more DBoxes.
- DBox removal cannot be performed at the same time when DBox migration is running (even if it is for a different DBox).
- DBox removal cannot be performed at the same time when DBox replacement is running.
- During DBox removal, it is not allowed to start a DBox expansion process. Adding or removing CBoxes is allowed.
- During DBox removal, the Write Buffers on Flash functionality is disabled.
- DBox removal is not supported after a Write Buffer RAID rewrite.
- If the cluster has DBox HA enabled, removing a DBox is not allowed if it would reduce the cluster below the minimum number of DBoxes required for DBox HA.
- If the cluster has rack resiliency enabled, the DBoxes to be removed must be balanced across all failure domains so that the removal does not impact failure domain balancing. Alternatively, the DBoxes to be removed must constitute a single failure domain.

DBox Type	DBox HA enabled	DBox HA disabled
Ceres	15	4
Mavericks	22	4

Callhome and Support

An attempt to delete a support bundle before it is completely created may cause unexpected behavior.
ORION-337635: In case of VMS/leader movement at the time prior to support bundle collection, the support bundle may include logs from both old and new nodes hosting the service. However, old nodes are not indicated as such in the log filenames. To differentiate between old and new nodes, use the timestamps within the logs.

Known Issues

Install and Upgrade

ORION-394488: When upgrading a cluster with block volumes from 5.4.x to 5.5.x, the existing volumes may be shown with a namespace ID of 0 instead of their actual namespace IDs during the upgrade. When the upgrade completes, the correct namespace IDs are shown for all block volumes.
ORION-361912: In VAST Cluster Install, skipped checkpoints are displayed as passed in the Checkpoint tab. The correct checkpoint status (Skipped) is shown in the Full Log tab.

Cluster Expansion

ORION-220738: In some cases, VMS does not provide any alerts or other status indication when a drive gets disabled while the newly added DBox is being initialized.
ORION-175762: In some cases, a DBox expansion procedure run on a cluster with similarity-based data reduction enabled can take longer than expected.

Multi-Tenancy

ORION-279664: The names of customized analytic reports created by a tenant admin, can be seen by other tenant admins. This issue affects only the report names; the data within the reports are not accessible from other tenants.

Quotas

ORION-387587: A 503 Forbidden error is returned when a tenant admin tries to create or edit user quotas on directories within the tenant, regardless of the permissions configured for the tenant admin manager user in the VMS.

Quality of Service (QoS)

ORION-337326: QoS burst metrics calculated for views or volumes subject to view/volume QoS policy limits, may in some cases show incorrect values.
ORION-139913: When applying a QoS policy to NFSv3 access, both data and metadata are taken into account in QoS limit calculations, while with NFSv4.1, only data are considered.
ORION-137986: Enabling a QoS policy for a view on which a mixed (read and write) workload runs, can result in decreased performance for the workload.

SMB

ORION-402172: Sporadic SMB disconnects may be encountered during SMB server-side copy.
ORION-395371: SMB clients may in some cases encounter intermittent access denied errors. The issue would typically occur in a multi-domain environment when a user is a member of multiple groups belonging to a domain that is different from the user's home domain, and such user performs a SMB operation permissions for which are granted through one of these groups.
ORION-142968: If a quota is exceeded during the process of coping a file to the VAST cluster, the copying process is stopped with a misleading error message: A device attached to the system is not functioning.

S3

ORION-378803: When responding to a POST upload request with no object key, the cluster returns error code 400 followed by InvalidObjectName instead of InvalidArgument.

Attribute-Based Access Control (ABAC)

ORION-196170: When a parent and child NFSv4.1 view both have same ABAC tags on them, an attempt to mount the child view may result in a permission denied error. If this occurs, try setting the ABAC tags for the machine account that the client uses to mount the view.

VAST Audit Log

ORION-393002: The VAST Audit Log page in VAST Web UI (DataBase -> VAST Audit Log) may show raw values in some fields. These values are taken from the cluster's internal database and can be negative for some S3 operations. For example, the Path/Handle ID field will show a negative value for a CreateMultipartUpload operation since there is no active handle associated with the operation.
ORION-360698: VAST Audit Log records for presigned POST requests specify the request type as PUT_OBJECT.

VAST DataEngine

ORION-410154: The VAST Web UI dialog used to create a managed application (Data Engine -> Applications -> choose to create or edit an application -> Resource Selection) provides incorrect guidance on the number of CNodes required for a managed application.
ORION-402194: A DR_IS_NOT_ACTIVE view_create returned an error: ObjectResultCode: INVALID_PARAM error may occur on clusters running VAST DataEngine with data reduction disabled.
ORION-389933: Scheduled triggers cannot start pipelines while the VMS container is not running (which may occur during NDU or HA events). If the VMS is down at the time when a pipeline is about to be triggered by a scheduled trigger, the pipeline will not start. After the VMS is back up, the pipeline will be triggered as expected. Note that the issue occurs with scheduled triggers only. Element triggers work regardless of the VMS container state.
ORION-383456: When removing a tag from a trigger in the Update Trigger dialog (opened from the Manage Elements -> Triggers page), the tag does not get removed from the trigger in the backend. Use VAST DataEngine CLI or API to remove tags from triggers.
ORION-379170: The Tenant dropdown on the Data Engine -> Compute Clusters page may take a long time to load the list of tenants.
ORION-358149: Inappropriate combination of batch event timeout and function timeout settings may result in duplicate entries appearing in the batch event traces. The function timeout must be set to a value exceeding the batch timeout (600 seconds by default).

Replication

ORION-393875: An attempt to add a third member to a replication group may got stuck if the two destination peers in the group run different VAST releases, as follows:
- One peer is on version 5.4 or later, while the other is on a version earlier than 5.4.
- One peer is on version 5.1 or later, while the other is on a version earlier than 5.1.
ORION-391617: Simultaneously creating S3 buckets on 2 clusters configured to run synchronous replication with Bucket Replication enabled, where the bucket paths are being synchronously replicated to the other cluster, may fail. Either create both buckets on one cluster, or wait until the first bucket creation completes before starting the second.
ORION-140894: When attempting to delete a protected path from the destination peer after an ungraceful failover, a Failed to delete following streams or a similar error occurs. The workaround is to manually change the destination peer's role to STANDALONE and retry the deletion.

Global Access

ORION-145307: Bulk permission updates are not supported for files and directories on satellite clusters.

Authentication and Authorization

ORION-371895: In some cases, when using LDAP groups for authentication when logging in to the VMS, the login may fail if the user is a member of more than 440 groups.
ORION-353874: A Setting mtls certificate for protocol NFS failed: CA certificate chain is empty error occurs when trying to upload a trust chain to tenant's NFS mTLS settings (Element Store -> Tenants -> choose to create or edit a tenant -> Advanced Protocol Settings tab -> Authentication settings for mTLS pane) where the root CA is not the first in the chain.
ORION-25479: Latin characters are not supported with LDAP. If you attempt to pass, for example, a username encoded with a Latin character set, the LDAP sanity check res: Invalid credentials error is returned.

VMS

ORION-401264: The VAST Prometheus Exporter endpoint /api/prometheusmetrics/user_view may produce duplicate metric entries. If you encounter this issue, contact VAST Support for a workaround.
ORION-398271: The Block Volume IOPS by Size metrics show 0 in case there are less than 15 operations within a 15-second time slot.
ORION-300508: Some metrics may not be available in VAST Prometheus Exporter. Missing metrics include (but may be not limited to) CNode and DNode hardware metrics or CPU temperature, utilization, memory percentage, PCI error count, retransmitted segments, total correctable/uncorrectable memory errors. If you encounter this issue, contact VAST Support for a workaround.
ORION-292325: When creating a customized analytics report for block volumes or hosts, some metrics intended to reflect a maximum latency are set to a fixed value when there are no block volumes on the cluster.
ORION-143717: On a cluster with CNode Port Affinity configured, there is no way to expose the VAST DNS IP on a specific port (left or right).
ORION-131386: When there is a parent directory that has a very large number of child directories, a total of children’s capacity values displayed in the Capacity page can exceed the capacity value shown for the parent directory.
ORION-89570: In some cases, capacity analytics for subdirectories cannot be reported due to an internal timeout. This issue occurs when there is an extremely large number of subdirectories to be estimated.

VAST Web UI

ORION-403079: An Invalid Argument error can occur when trying to add a tag to a snapshot volume (Element Store -> Volumes -> choose to edit a volume).
ORION-399000: The Rewrite Status popup (appears after you initiate a data rewrite on the cluster) may sometimes show decreasing progress percentage due to discovery of additional items that are subject to the rewrite.
ORION-396255: The VMS does not display LED status for Turin CNodes. The corresponding field in the Infrastructure -> CNodes page shows Unknown or N/a or is always Off.
ORION-396165: When Kafka is selected in the Performance pane on the cluster's dashboard, the Latency popup has an extra field named Avg that is always 0. This field can be ignored.
ORION-395373: When viewing top actors among volumes (Analytics -> Top Actors -> Volumes selected), the right-side Top Actors pane may show entries with N/A instead of the volume name. Such entries do not represent real volumes and can be ignored.
ORION-394425: In the Analytics -> Data Flow page, if the Sort field is set to BW, MD IOPS or Latency, selecting the Scan option in the By field causes the page to hang. Use the Scan option only when the Sort field is set to IOPS or ROW/s.
ORION-379170: The Tenant dropdown on the Data Engine -> Compute Clusters page may take a long time to load the list of tenants.
ORION-371787: The Default Gateway field in managed application settings (Data Engine -> Applications -> choose to create or edit an application -> Network tab -> Advanced pane) does not work as expected.
ORION-308654: When using the Export selected rows as CSV option of the VAST Web UI, the resulting file may contain all of the rows, including those that were not selected for the export.
ORION-294888: The Infrastructure pages of VAST Web UI may display [object Object] instead of the actual CPLD/UBM values for Dell Turin CNodes.
ORION-234835: Some VAST Web UI pages might not allow for proper filtering or sorting by column where value presentation differs from that in the VAST internal database.
ORION-203189: The External Netmask field in cluster networking settings (Settings -> Configure Network) does not accept alphabetic characters.

VAST CLI

ORION-363096: When associating CNodes with a virtual IP pool using the vippool create or vippool modify command, there is no validation of whether the CNode count is sufficient, which could result in creating a pool that lacks the required number of CNodes.
ORION-322120: When displaying path names in VAST CLI, the names written in right-to-left languages may appear spelled from left to right.
ORION-265720: VAST CLI auto-completion does not include the --detach-krb-provider option on the tenant modify command.

VAST REST API

ORION-178569: The /users/names endpoint always returns only the first 50 entries, regardless of the page size parameter or the total amount of entries to be returned.

Platform and Control

ORION-403644: If an administrative role has OS SSH login enabled (Administrators -> Administrative Roles -> select a role -> Enable OS SSH Login option), an attempt to enable OS SSH login for a second role may result in an internal timeout, which also causes the feature to stop working for the first role.
ORION-255054: Once a DNode replacement operation is complete, the old node may still be listed in the Infrastructure -> DNodes page of VAST Web UI for some minutes.

Callhome and Support

ORION-269349: When obfuscating a support bundle, the obfuscation is not applied to files that have already been zipped.
ORION-239170: When obfuscating a support bundle, the CNode hostname may not get obfuscated in some of the logs included in the bundle.