Understanding VAST Capacity

Introduction

As with any storage system, capacity is an important concept to understand. In VAST, there are three types of capacity to consider: logical, physical, and usable. Each will be explained in depth, but briefly:

• logical capacity - How much data can be written to VAST? This is VAST's prediction of the amount that can be written to the system. This amount of data is what applications actually write. Basically, if an application thinks it wrote 1GB, that is the number. VAST adjusts the logical capacity by considering various overheads and data reduction.

• usable capacity - how much data can be written to VAST, assuming no data reduction. This number is how much data can be written to VAST after considering VAST overheads (such as data protection) but not considering data reduction. Another way of thinking about this number is this: if data reduction is none, then logical capacity equals usable capacity. If there is some data reduction, the logical capacity is larger than the usable capacity.

• physical capacity - the literal physical storage space reported by the drives after an initial 8% space reservation on system install. The amount of data that can be written to VAST if there is no data reduction (aka usable capacity) is always less than this amount because of the overhead introduced by VAST.

Until version 3.6, VAST reports physical capacity and logical capacity but not usable capacity. The product documentation explains the distinction between logical and physical capacity in the Dashboard, but that may leave some questions unanswered. In this article, we go into a bit more depth on physical vs. usable capacity and how that relates to logical capacity.

Usable Capacity vs. Physical Capacity

When considering usable capacity, one starts with the raw physical capacity of the underlying physical storage and then subtracts overhead:

• Data Protection - VAST uses large erasure coding/RAID stripes to protect data. Parity blocks introduce overhead. The data protection overhead varies by the size of the erasure coding stripe, which in turn varies based on the number of storage enclosures (D-boxes) in a cluster. This overhead is shown incrementally as new data is written by increasing the physical space used by more than was written after data reduction. The overhead ranges from 2% to 11% as VAST varies the number of data blocks in the stripe from 36 to 146; the number of parity blocks is always 4. In this article, we will write that as D+P, where D is the number of data blocks (36-146), and P is the number of parity blocks (4) per stripe. P is the overhead.

• Reserve - While VAST is not a log-based file system in the traditional sense, VAST needs to keep a certain amount of physical space free in order to enable its background activities. The reserve will decrease with each VAST version.  Eventually, our objective is to reduce the overhead to just 8%.  In keeping with that objective, the reserve is shown in two parts - at initial cluster bringup by reducing physical capacity shown by 8%, and over time by increasing the physical space used by each write by an additional percentage that depends on the release.  The second value is shown incrementally because the goal is to reduce it to zero eventually. The release-specific additional percentages are:

  • 3.0.x: 3% for a total of 11%

  • 3.2.x and later: 1% for a total of 9%

The fixed initial space reservation can be easily shown as in this diagram:

This is what is shown as the physical space upon cluster initialization. Please keep in mind that VAST, prior to VAST 3.6, does not report usable capacity. Beginning with VAST 3.6, VAST will report usable, physical, and logical capacity. The tables below are to help you plan how much you can store on a VAST system.

In tabular form, here is what the above means in terms of usable capacity - assuming no data reduction:

Note: Half-populated storage enclosures have slightly different overhead values, and clusters where D-box HA / rack-level protections are enabled have higher overhead values. Consult your VAST representative for the specifics.

Logical Capacity

The actual amount that can be written depends heavily on Data Reduction. Data reduction reduces the amount of usable capacity consumed by data, which results in an increase in the logical capacity available for the data.  VAST uses three techniques for data reduction: global compression, global deduplication, and, very soon, global similarity compression. Data reduction reduces the amount of data written to some amount below what is logically written.  Data reduction is reported in VAST and readily visible in the dashboard - the value reported does not consider the overheads mentioned earlier that will slightly increase what is written to disk. In most cases, data reduction exceeds the overheads by a large margin.

Logical capacity is a prediction of how much data can be stored, assuming similar future data reduction ratios and no change in the overheads. Logical capacity is a much better predictor over time of the amount of data that can be stored in VAST than physical or usable capacity. The only caveat is that one must first store enough representative data into the system before the logical capacity prediction is meaningful.

The following diagram shows the relationship between logical and physical. The most important takeaway is that the total logical space (aka logical capacity) is calculated dynamically based upon the actual logical used, physical used, and total physical space.  Essentially, the ratio of logical used to total logical space must always be the same as physical used to total physical space. Logical free is adjusted accordingly: logical free = logical space - logical used.

Example

The following diagram summarizes the VAST behavior with respect to the initial space reservation of 8%, data reduction, erasure coding, and incremental space reservation (1% today) for a single 1GB write:

Consider an example that should make this more concrete. Suppose a cluster contains four 675TB storage enclosures. The total physical space in VMS per enclosure will be reported as 92% of 675TB = 621TB - as mentioned earlier, that is the initial space reservation for internal overhead. Obviously, for four enclosures, the space reported is 621*4 = 2484TB. Then, when a write of 1GB occurs, we will assume that it can be reduced by 50% using data reduction. The following will happen, assuming a cluster with at least four full-sized storage enclosures (enabling 146+4 stripes):

• The application logically writes 1GB, andthe  standard file system APIs report that 1GB was written

• Data reduction will reduce the actual data to be written to 500MB

• Erasure coding overhead will add in 2.8% (4/146) more physical space to be used

• Reserve overhead will add in 1% (as of 3.2) more physical space to be used

• Total bytes written is then 500MB x (100% + 2.8% + 1%) = 519MB

• Physical space used will increase by 519MB, and physical space free will decrease by 519MB

• Logical space used will increase by 1GB

• Logical space free will be adjusted to reflect the new predicted logical storage space remaining.

Quotas

Quotas in VAST are against the logical space used. Thus, when an application logically writes 1GB of data, its quota usage increases by 1GB. This is done intentionally as usable or physical used is largely meaningless to end users. Consider that one user could write 1GB of data and get no data reduction, while the second user might write 1GB of different data and get a large amount of data reduction because a third user already wrote similar data. But at some future time, that third user might delete similar data - it would make no sense to change the second user's quota usage based upon what some other user did.

Two caveats regarding quotas:

• VAST internally tracks file sizes using 1-byte increments. This means that the smallest quota increment is 1k. Thus, a 1K file written to VAST will consume at most 1K of space (even less with data reduction), so the quota used is 1K.

• There is going to be a product enhancement to optionally enable more accurate quota accounting, but this will introduce the side effect of quotas not properly considering sparse files - essentially, the quota usage for a sparse file will consider its full logical size, not the bytes written.