Quality of Service (QoS) policies let you set limits to define maximum allowed bandwidth and/or IOPS for read and/or write operations, and also prioritize workloads. A QoS policy can be assigned to a view or to a user.
QoS Policy Types
A QoS policy can be intended to provision VAST Cluster performance for a view or for one or more users.
In a view QoS policy, you can set static and/or capacity-based QoS limits for a specific view.
After creating a view QoS policy, you can assign it to a view when creating or editing a view.
View QoS policies are supported for NFSv3, NFSv4, SMB and S3.
In a user QoS policy, you can set static QoS limits for one or more users.
After creating a user QoS policy, you can assign it to the user(s) when editing a user. A user can be assigned only one QoS policy per tenant.
User QoS policies are supported for NFSv3, NFSv4, SMB and S3.
Tip
Before creating user QoS policies, you need to enable the user QoS functionality on the VAST cluster by creating a default user QoS policy.
If an IO operation is subject to more than one QoS policy, it is handled as follows:
The IO operation consumes resources from all of applicable policies.
The IO operation is allowed to proceed only if it does not consume more resources than allowed by all the applicable policies.
For example, if a user has a limit of 1GB/s and is reading from a view that has a limit of 500MB/s, the user can only read at 500MB/s from that view. In addition, the user can read at 500MB/s from other views.
Static and Capacity-Based Limits
You can set static and/or capacity-based QoS limits.
Static limits are used-defined values of bandwidth and/or IOPS for read and/or write operations. These limits are fixed; they are not dependent on the used or provisioned capacity. Static limits can be set in view QoS policies and in user QoS policies, as well as for the entire cluster.
Maximum static limits define the maximum allowed performance per view or per user when there is no resource contention.
You can also set burst and credit static limits.
Static limits do not require a quota to be provisioned on the path or for the user.
Capacity-based limits can be based on either of the following:
Used logical capacity
These limits cap the read/write bandwidth and/or IOPS per unit of used logical capacity. (Logical capacity is the amount of data written, before data reduction.) For example, you can configure a QoS policy to allow a maximum of 3 read IOPS per one GB of used logical capacity.
Provisioned logical capacity
These limits cap the read/write bandwidth and/or IOPS per unit of logical capacity provisioned by the soft limit of a quota on the path. For example, you can configure the QoS policy to allow a maximum of 1MB/s of read bandwidth per GB of the capacity limit configured in a quota.
Capacity-based limits can only be set for view QoS policies.
In view QoS policies, capacity-based limits require a quota to be provisioned on the path. If you attach a QoS policy that sets capacity-based limits to a view that has no quota, VMS automatically creates a quota on the path. The quota on the path itself is the only quota that affects the limits.
QoS limits take both data workload and metadata requests into account, as follows:
The size of a single IO for the purpose of a QoS limit is 1MB. In other words, the number of IOs per request is obtained by dividing the request size by 1MB.
A mutating metadata request (such as create, delete, rename, setattr) is counted as one write IO. Non-mutating metadata requests (such as getattr, lookup, list) are counted as one read IO for each. The size of a metadata request is taken to be 4KB.
Burst and Credit Limits
You can set a burst limit and a credit limit.
When a burst limit is set, burst credits are accumulated as long as the workload consumes less resources than set by the maximum limit. The credits can later be spent to gain performance that exceeds the maximum limit, up to the configured burst limit.
The amount of credits that can be accumulated is capped by a credit limit set in a QoS policy.
For example, if a QoS policy defines a maximum limit of 100MB/s, a burst limit of 1000 MB/s and a credit limit of 10000, then after 100 seconds of idle time, the credit limit will be reached. Given that credit balance, the application can run at a 1000MB/s for 11 seconds following which it will be throttled down to 100MB/s.
If no credits are explicitly defined, setting a maximum static limit will cause the corresponding credits to accept a default value. The credit default value will be the maximum limit multiplied by 4. For example, if you set the maximum allowed read bandwidth to 500 and do not specify any value for the read bandwidth credit, the read bandwidth credit will automatically be set to 2000.
S3 Connection Limit
The S3 connection limit restricts the maximum number of S3 connections to the VAST cluster that can be opened by a client user. This helps prevent scenarios where a single client IP creates an enormous amount of S3 connections exhausting cluster's TCP connection resources.
When set in a QoS policy attached to a specific user or group, the S3 connection limit applies to that particular user or group.
When set in the default user QoS policy, the limit applies to any S3 user connecting to the VAST cluster.
Note
The S3 connection limit does not affect root users.
By default, this feature is enabled but no S3 connection limit is set.
A connection is attributed to a user only after the cluster has received the first request from that user. If multiple users share the same connection, this connection is attributed to the user which made the first request. When there is no first request yet, the user is considered to be unknown.
If the S3 connection limit is set for a user that has already opened some S3 connections, the existing connections are not affected by the new limit and are not taken into account when imposing the S3 connection limit.
Tip
For all users that have an associated QoS policy with this limit set to greater than zero, you can view the number of active S3 connections per user in user query results.
Total Limits
The QoS limits you set in a QoS policy can be either total or specific to the type of operations (read or write). Thus, you can restrict the bandwidth for write operations while leaving the read bandwidth unlimited, or you can set a total limit that will cap the total amount of read and write IOPS.
Total limits take metadata operations into account. For example, if you set a total IOPS limit to 1000, then at any given point in time, the sum of read, write and metadata IOPS cannot exceed 1000.
Tenant Limits
Tenant limits are capacity-based limits and static limits that apply to the tenant as a whole. These limits are configured directly in tenant settings, without the need to create a QoS policy. For details, see Creating Tenants.
Note
With VAST Cluster 5.4, block protocol operations are not subject to tenant limits.
Cluster Limits
Cluster limits define QoS for the entire cluster:
The maximum write bandwidth provided by the cluster
The maximum bandwidth provided for replication, Global Access and global snapshots
Cluster limits are configured in cluster settings and serve as the basis for prioritizing workloads, as detailed in Prioritizing Workloads.
Combining QoS Limits
When static limits are used together with capacity-based limits, they limit performance that can be reached while still within the capacity-based limits. For example, setting a 1GB/s maximum and 10MB/s per GB would mean that after reaching 100GB, the performance will no longer increase.
However, if a burst limit is set, the performance can go beyond the specified capacity-based limit. For example, if you cap the bandwidth with a maximum limit of 300MB/s and a burst limit of 800MB/s while having a capacity-based limit of 500MB/s, the performance will be capped at 800MB/s.
Prioritizing Workloads
VAST Cluster offers the following methods to ensure that workloads are served with the expected QoS:
Define a cluster-wide maximum write bandwidth based on which workload prioritization will take place. This helps prevent situations where workloads do not achieve the expected QoS because of extensive media consumption by other workloads.
The recommended cluster-wide maximum is 70% of the cluster’s total write bandwidth.
To define a cluster-wide maximum write bandwidth:
In VAST Web UI, go to the Global Write BW Limit pane in cluster settings (Settings -> Cluster -> General Cluster Setup and Actions tab). Select Set Manually and enter the write bandwidth limit in the provided fields.
In VAST CLI, run the
cluster modifycommand with the--max-cluster-write-bw-mboption specified.
Set the prioritization flag for the QoS policy that controls high-priority workloads. This will prioritize the workloads in contention for both CPU and memory resources beyond the limits defined for the tenant and/or for the cluster.
Note
The prioritization flag is supported for view QoS policies. It cannot be set for user QoS policies.
To set the prioritization flag for a QoS policy:
In VAST Web UI, enable the Prioritize policy over cluster or tenant limitations option in QoS policy settings (Element Store -> QoS Policies -> choose to create or edit a QoS policy -> View tab).
In VAST CLI, run the
qospolicy createorqospolicy modifycommand with the--is-goldoption specified.
Tip
Since prioritized workloads are not restricted by the tenant or cluster-wide limits, it is recommended to set a maximum limit for such workloads within the QoS policy.
The following rules and restrictions apply:
The cluster-wide maximum write bandwidth limit affects NFS and SMB workloads only. Other access protocols are not supported.
When the cluster-wide maximum write bandwidth is set, it cannot be exceeded even when QoS credits exceed the cluster-wide limit.
When one or more views are control with QoS policies for which the prioritization flag is set, non-prioritized workloads may experience up to 20% performance degradation. The degradation may occur even when the cluster-wide maximum write bandwidth limit is set.
Monitoring Performance with QoS Policies Enabled
When using VAST Web UI dashboard and analytic reports to monitor performance of a VAST cluster that has one or more QoS policies in effect, it is recommended to set the UI to use base2 units of measure. This ensures more accurate alignment between QoS calculations, which are in base2, and the performance metrics representation in VAST Web UI.