Capacity Analysis with vastpy-cli

Summary

This document provides a comprehensive guide on how to analyze storage capacity and file data on a VAST Data platform using vastpy-cli. We cover both the direct capacity queries and the more advanced catalog-based analysis.

Methods

Via VAST REST API

We will cover two methods of obtaining this data.  The first, using the VAST Capacity Estimations data, leverages the VAST REST API (via the vastpy-cli utility) to query the capacity statistics for any directory within the VAST Platform.  This API is available in all versions of VAST from 4.0 onward and requires no configuration.  For more information, see Capacity Overview.

This approach is useful if:

1. The VAST Catalog is not enabled on your cluster

2. You want insight into the DataReduction ratio across different folders on the system.

Via VAST Catalog

The Second approach is to use vastpy-cli to query the VAST Catalog, a feature that has been available since the 4.6 release.  This does require that the VAST Catalog is enabled; see VAST Catalog Overview for more details on the feature, as well as how to configure.  This approach is preferred if:

1. The VAST Catalog is enabled

2. You want more granular insight and filtering capabilities.  Examples include:

   • Determine capacity used per user/group

   • Determine capacity per file extension

   • Find individual file locations based on a filename search

   • Much, much more.

Regardless of which of these approaches you choose, you will be using the same basic tool (vastpy-cli).  There are other methods of accessing this data, which will be covered in another document.

Prerequisites

A host with:

• Python installed

• Network access to the VMS IP address, specifically on port 443 (HTTPS)

• VMS Manager account with appropriate permissions

Setting Up vastpy-cli

Installing vastpy

Install the vastpy package, which includes the vastpy-cli command:

pip install --upgrade vastpy

You may also want to install the jq utility:

sudo apt-get install -y jq #ubuntu/debian variants
sudo yum install -y jq #rocky/rhel variants

Setting Up Authentication

Set up environment variables for authentication (recommended method):

# Set these environment variables once per session
export VMS_USER=admin
 export VMS_PASSWORD=your_password  # Add a space before this command to avoid storing the password in command history
export VMS_ADDRESS=vast-vms-address

Alternatively, you can specify credentials with each command:

vastpy-cli --user=admin --password=your_password --address=vast-vms-address [command]

1. Analyzing Directory Capacity with vastpy-cli

The capacity subcommand in vastpy-cliallows you to retrieve detailed capacity information for specific directories in your VAST Data platform.

Generally, you filter by specifying the path you want to analyze.  It can be a top-level directory or any subdirectory within the system.  Note that the data underpinning this API is an estimate based on sampling and should not be used for exact calculations.  Also, there may be a delay between when data is ingested into a folder and when it appears in the statistics.  This is because sampling can occur as the data is migrated from SCM -> QLC flash, which occurs asynchronously.  Lastly, folders that do not have enough aggregate capacity will not appear on their own in these metrics.  If you are unsure of the information you are seeing, contact CS with specific commands and output so they can best assist.

Basic Usage

To get capacity information for a specific directory:

# Basic capacity command
vastpy-cli get capacity path=/path/to/directory

Retrieving JSON Output

For more structured output and easier parsing, you can request JSON format:

# Get capacity information in JSON format
vastpy-cli get --json capacity path=/path/to/directory

Example with jq Parsing

You can pipe the JSON output to tools like jq for further filtering and analysis:

# Get available capacity keys
vastpy-cli get --json capacity path=/maria | jq '.keys'

Output:

[
  "usable",
  "unique",
  "logical"
]

To get detailed information about the directory:

# Get detailed capacity information for the directory
vastpy-cli get --json capacity path=/maria | jq '.details[0]'

Output:

[
  "/maria",
  {
    "data": [
      7546623507,
      5237582703,
      25589375620
    ],
    "parent": "/",
    "percent": 100,
    "average_atime": "2025-03-29 07:30"
  }
]

(The output above is in bytes)

Understanding Capacity Output

The capacity output contains three key metrics:

1. Usable - The usable space in bytes on the physical media

2. Unique - The unique space in bytes (after deduplication)

3. Logical - The logical space in bytes (what users/applications see)

The average_atime field shows the average last access time for files in the directory, which can be helpful for understanding data usage patterns.

2. Analyzing Data with VAST Catalog

The VAST Catalog provides a more powerful way to query and analyze your data, especially for large datasets. You can use the bigcatalogconfig subcommand in vastpy-cli.

Some notes:

1. Depending on the filter and the number of files on the system, these queries can take some time to complete.  Most should complete within seconds, but some may take longer

2. There are no built-in 'aggregation' functions in this API at the moment.  If you wish to determine the capacity of a folder, for example, you have the following options:

   • Pipe through jq (described in some of the examples below)

   • Use another method (such as the vastdb-sdk), as described in the document here: VAST DB Python SDK .

   • Use an application such as Trino, as described here: Trino Quickstart

3. limit is a preview control, not a pagination control.  Use limit=50 when first experimenting with a query so it doesn't overrun your terminal.  Be aware of two consequences: any aggregation you run with a limit in place will be computed over the truncated rows and will therefore be wrong, and setting limit forces the response's next field to null, which disables paging entirely.  To retrieve a complete result set, drop limit and page through the results as described in Retrieving Complete Result Sets.

4. Always pass --json.  The default table output prints rows only and discards the count and next fields, which are the fields that tell you whether more results exist.

Using vastpy-cli for Catalog Queries

First, an example to list all files that were not accessed in the last 90 days:

# Basic catalog query for files not accessed in the last 90 days
vastpy-cli post --json bigcatalogconfig/query_data \
  limit=50 \
  path="/vperfsanity" \
  fields='["parent_path", "name", "atime", "size"]' \
  filters='{"atime": [{"lt": "'$(date -d '90 days ago' +%s)'000"}]}' | jq

Output (truncated):

{
  "count": 256,
  "next": null,
  "prop_list": [
    "parent_path",
    "name",
    "atime",
    "size"
  ],
  "results": [
    [
      "/vperfsanity/",
      "vperfsanity21",
      "2025-01-14T20:14:49.499298",
      4194304000
    ],
    [
      "/vperfsanity/",
      "vperfsanity155",
      "2025-01-14T20:14:47.415081",
      4194304000
    ]
  ]
}

Same query, but this time only retrieve the size field and perform a sum (notice that we removed the limit from this example to get an accurate sum.

vastpy-cli post --json bigcatalogconfig/query_data \
  path="/vperfsanity" \
  fields='["size"]' \
  filters='{"atime": [{"lt": "'$(date -d '90 days ago' +%s)'000"}]}' | jq '([.results[][]] | add)'

Output:

1073741824000

Obviously, you may want to convert from bytes into something more useful, like GiB or TiB:

vastpy-cli post --json bigcatalogconfig/query_data \
  path="/vperfsanity" \
  fields='["size"]' \
  filters='{"atime": [{"lt": "'$(date -d '90 days ago' +%s)'000"}]}' | jq '([.results[][]] | add) / (1024*1024*1024)'

3. Retrieving Complete Result Sets

Every example above returns a single response.  When a query matches more rows than the server returns at once, the response carries a next field holding a URL ending in ?page=N.  Paging means re-issuing the same query with an added page field until next comes back null.

vastpy-cli is a thin wrapper over the REST API, so the paging rules are the API's rules. Two CLI behaviors shape how you apply them:

• POST puts every argument in the JSON body; GET puts them on the query string.  There is no way to mix a query-string page with body filters in one invocation.

• --file-input replaces key=value arguments; the two cannot be combined.  If you supply a JSON file, any key=value arguments on the same command line are ignored silently.  For page 2 and beyond, put page inside the JSON file.

The two rules that matter

1. Never send limit together with page.  The combination returns a 400 with an empty detail body.

2. Never send limit at all when you intend to page. limit on its own forces next to null, so you cannot tell whether more rows exist.

Recommended: POST with --file-input

Request the first page with no limit and no page:

export VMS_USER=admin
export VMS_PASSWORD='***'
export VMS_ADDRESS=vast-vms-address
export VMS_API_VERSION=v5   # or latest

cat > catalog_query.json << 'EOF'
{
  "path": "/",
  "snapshot": "bc_table_2026-09-08_10_00_00",
  "fields": ["parent_path", "name", "size", "mtime"],
  "filters": {"size": [{"ge": 1048576}]}
}
EOF

vastpy-cli post --json bigcatalogconfig/query_data --file-input catalog_query.json

Inspect next in the response.  If it is a URL containing ?page=2, wait for the call to finish, then add "page": 2 to the same file, leaving everything else unchanged:

{
  "path": "/",
  "snapshot": "bc_table_2026-09-08_10_00_00",
  "fields": ["parent_path", "name", "size", "mtime"],
  "filters": {"size": [{"ge": 1048576}]},
  "page": 2
}

vastpy-cli post --json bigcatalogconfig/query_data --file-input catalog_query.json

Repeat with "page": 3, "page": 4, and so on until next returns null.

Pinning snapshot to a specific catalog table ensures a consistent multi-page walk for the lifetime of the query.  The query will work without it, but it is not recommended when point-in-time accuracy matters.

Paging constraints

The server enforces the following.  Violating any of them produces skipped or duplicated rows rather than an error:

Looping through every page

page=1
cp catalog_query.json /tmp/q.json   # no page, no limit

while true; do
  if [ "$page" -gt 1 ]; then
    jq --argjson p "$page" '.page = $p' catalog_query.json > /tmp/q.json
  fi
  vastpy-cli post --json bigcatalogconfig/query_data --file-input /tmp/q.json \
    > /tmp/page.json
  count=$(jq '.count' /tmp/page.json)
  next=$(jq -r '.next // empty' /tmp/page.json)
  echo "page $page: $count rows  next=${next:-null}"
  [ -z "$next" ] && break
  page=$((page + 1))
done

GET form

GET sends parameters as a query string, which VMS also accepts.  Keep limit out and keep --json.  Note that filters supplied as CLI JSON is parsed by vastpy into a dict and can serialize badly, so prefer the POST --file-input form whenever filters are nested:

vastpy-cli get --json bigcatalogconfig/query_data \
  path=/ \
  fields=parent_path,name,size,mtime

vastpy-cli get --json bigcatalogconfig/query_data \
  path=/ \
  fields=parent_path,name,size,mtime \
  page=2

Python SDK

For anything scripted, the SDK is cleaner than the CLI.  POST keyword arguments become the request body, so page is simply another field:

from vastpy import VASTClient

client = VASTClient(address='vast-vms-address', user='admin', password='***', version='v5')
body = {
    'path': '/',
    'snapshot': 'bc_table_2026-09-08_10_00_00',
    'fields': ['parent_path', 'name', 'size', 'mtime'],
    'filters': {'size': [{'ge': 1048576}]},
}

page = 1
while True:
    kwargs = dict(body)
    if page > 1:
        kwargs['page'] = page
    resp = client.bigcatalogconfig.query_data.post(**kwargs)
    print('page', page, 'count', resp['count'], 'next', resp.get('next'))
    if not resp.get('next'):
        break
    page += 1

Advanced Query Examples

The examples below use limit=50 to keep the output readable.  Remove limit and page through the results, as described above, whenever you need the complete set.

Finding Large Files

# Find files larger than 1GB
vastpy-cli post --json bigcatalogconfig/query_data \
  limit=50 \
  path="/vperfsanity" \
  fields='["parent_path", "name", "size"]' \
  filters='{"size": [{"gt": 1073741824}]}'

Finding Recently Modified Files

(Note the use of gt as opposed to lt)

# Find files modified in the last day
vastpy-cli post --json bigcatalogconfig/query_data \
  limit=50 \
  path="/" \
  fields='["parent_path", "name", "mtime"]' \
  filters='{"mtime": [{"gt": "'$(date -d '1 day ago' +%s)'000"}]}' | jq

Finding Files by Owner

# Find files owned by a specific user
vastpy-cli post --json bigcatalogconfig/query_data \
  limit=50 \
  path="/" \
  fields='["parent_path", "name", "login_name"]' \
  filters='{"login_name": "maria.gutierrez@selab.vastdata.com"}'

Find Files by filename

vastpy-cli post --json bigcatalogconfig/query_data \
  limit=50 \
  path="/" \
  fields='["parent_path", "name", "login_name"]' \
  filters='{"name": "comment.mp4"}'

Output

{
  "count": 2,
  "next": null,
  "prop_list": [
    "parent_path",
    "name",
    "login_name"
  ],
  "results": [
    [
      "/scratch/home/rayc-flow/MusM/",
      "comment.mp4",
      null
    ],
    [
      "/scratch/home/jhays-flow/feasibleness/slim/",
      "comment.mp4",
      null
    ]
  ]
}

Tips for Large-Scale Analysis

When working with large datasets:

1. Filter Early - Apply specific filters to reduce the dataset size

2. Page, don't limit - limit truncates the result and suppresses next.  For a complete result set, drop limit and walk the pages consecutively, pinning snapshot so the view stays consistent.  See Retrieving Complete Result Sets.

3. Aggregate over every page - Any jq sum or count must accumulate across all pages.  Aggregating a single page silently under-reports.

4. Schedule Regular Analysis - Set up regular analysis jobs to track capacity trends over time

Conclusion

The VAST capacity and catalog analysis tools provide powerful ways to understand your data usage patterns, identify optimization opportunities, and manage your storage more effectively. By combining direct capacity queries with catalog analysis, you can gain comprehensive insights into your VAST Data platform.

For more detailed information about the VAST API, please refer to your VAST system's documentation at https://<your-vast-vms-address>/docs.