Querying the VAST Catalog

A guide to querying the VAST Catalog — the cluster's built-in, queryable index of every file, directory, object, table, and symlink — through the database. It starts with the path that works with nothing but VMS credentials, then adds the VAST DB SDK for programmatic access.

Scope. This guide covers database access to the catalog: the VMS REST API and the
VAST DB SDK (plus the Web UI where your version exposes a catalog grid). External Trino
query engines and Row and Column Security are out of scope (Section 5), and so is point-in-time
querying of catalog snapshots. For the audit log — the other built-in queryable table,
which records what happened rather than what exists — see the companion guide Querying
the VAST AuditDB.

This is a getting-started guide. The official documentation is the reference for concepts, the full schema, and Web UI procedures — start with the VAST Catalog Overview;  the other official pages are linked from the sections here that provide overviews (Section 0, Section 2) and are  collected in Section 6.

What is the VAST Catalog?

The VAST Catalog (also called the Big Catalog) is a VAST Database table the cluster maintains about its own namespace: one row per element (file, directory, object, DB table, symlink...), with its path, type, size, ownership, timestamps, and S3 user tags/metadata. Instead of walking a filesystem over NFS or listing buckets over S3, which can be slow and inefficient, you query the index: "the 100 largest files under /projects", "everything owner X changed this week", "every object tagged retention=forever", each as one query.

The catalog is an index, not a live view: the cluster refreshes it on a schedule you configure (typically every 15–30 minutes, Section 0), so very recent changes appear after the next
refresh.

Ways to query it

The VMS REST API reads the catalog through the management plane and works the moment the catalog is enabled — and it sees the whole cluster, every tenant. The SDK reads the same table via the S3/tabular data endpoint and is automatically scoped to the querying key's tenant (see Section 5, Tenant scoping is automatic). Start with Section 2 for a quick look; use the SDK for anything programmatic.

Read in this order

Runnable starters, in the same easy-to-expert order: examples/query_vms_catalog.py (VMS API path), examples/first_query.py (SDK first steps: list a directory, --types, --schema), and examples/catalog_report.py (SDK: largest files, by owner, changed-since, by extension, tagged elements).

The 60-second mental model

1. Turn the catalog on and pick a refresh cadence (Section 0).

2. Query it right away through the VMS REST API — bigcatalogconfig/query_data, with apath subtree anchor and at least one filter (Section 2).

3. For SDK access, give the user an identity policy with the tabular read actions (s3:TabularList*, s3:TabularQueryData) and open the table with tx.catalog() (Section 5).

4. Always cap the read. Every SDK query is a scan over a table with one row per element on the cluster; limit_rows both cap the fetch and let the scan stop early, and low-match queries pay a full sweep (~15 s on a 3-billion-row lab catalog) no matter how selective the predicate looks (Section 3, Error 3).

Coming from the AuditDB guide?

The access mechanics are identical — same managed-bucket pattern, same identity policy, sametx.<well-known-name>() open — but three semantics flip:

Conventions

• The catalog lives at fixed, well-known coordinates on every cluster:vast-big-catalog-bucket / vast_big_catalog_schema / vast_big_catalog_table. The SDK has them built in (tx.catalog()), so most code never spells them out.

• Version-sensitive values (like the element_type list) are called out in place and collected in the verification checklist in Section 6.

• "Identity policy" and "S3 policy" are the same object in VAST (the s3policies API resource).

• Cross-references like "Section 3" or "Recipe 5" are deliberately plain text, not hyperlinks: these files get pasted into GitLab snippets and tickets, where relative links between files break. The only live links are absolute https:// URLs and anchors within the same file.

Code examples

The runnable example scripts this guide refers to — query_vms_catalog.py (VMS REST API path), first_query.py, and catalog_report.py (VAST DB SDK) — live in a companion snippet, with their own README covering setup and a quick tour:

0. Enable the Catalog

This page turns the VAST Catalog on and sets how often it refreshes. On many clusters it is already enabled — check first; it takes one call. (The official reference for this area, including the Web UI procedure, is Configuring VAST Catalog.)

The catalog is driven by one config object, the Big Catalog policy (bigcatalogconfig in the VMS API). It has two knobs that matter:

1. enable — whether the cluster maintains the catalog at all.

2. frames — the refresh schedule: how often the cluster snapshots its namespace into the catalog table (every), and how long those catalog snapshots are retained (keep-local / keep-remote). The every interval is your data freshness: an element created a moment ago appears in query results after the next frame.

If you haven't used the vastpy-cli before, check out this KB article.  Getting Started with vastpy-cli

Check whether it's already on

pip install vastpy
vastpy-cli --address <vms-host> --user admin --password '********' get bigcatalogconfig

Healthy output looks like this (from a VAST 5.4.4 lab cluster):

[{
  "id": 1,
  "name": "Vast Catalog Policy",
  "enable": true,
  "state": "working",
  "big_catalog": true,
  "prefix": "big_catalog",
  "clone_type": "LOCAL",
  "frames": [{"every": "30m", "start-at": "2026-04-28 20:01:25",
              "keep-local": "1H", "keep-remote": "30m"}]
}]

enable: true plus state: "working" means the catalog is being maintained and is currently queryable — skip to Section 2.

Enable it (or change the cadence)

Patch the policy by its id from the check above (typically 1). This example enables the catalog with a 15-minute refresh — reuse the start-at value the GET returned so the schedule anchor doesn't move:

vastpy-cli --address <vms-host> --user admin --password '********' \
  patch bigcatalogconfig/1 \
  enable=true \
  frames='[{"every": "15m", "every_value": "15m", "every_unit": "m",
            "start-at": "2026-04-28 20:01:25",
            "keep-local": "2H", "local-period": "H",
            "keep-remote": "15m", "remote-period": "m"}]'

(vastpy-cli parses values that look like JSON as JSON, so the frames list goes through as a real structure. If you'd rather keep the body in a file, the same call is vastpy-cli ... patch bigcatalogconfig/1 -i body.json.)

Then re-run the check above and confirm enable: true and state: "working".

Cadence trade-off: a shorter every means fresher query results and more frequent indexing work on the cluster. The cluster enforces a 15-minute minimum (the default is 30), and 15–30 minutes is the common range; whatever you choose, remember it when interpreting results — "no rows for the file I just wrote" usually just means "next frame hasn't run yet" (Section 3, 2c).

The same policy can be managed from the Web UI's Settings on versions that expose it — the field names match the JSON above. The official Configuring VAST Catalog guide documents that path, plus two things this guide deliberately doesn't cover: indexing user-defined attributes (S3 tags/metadata as extra catalog columns) and monitoring indexing progress via restore points. This guide uses the CLI because it is consistent across versions; see the verification checklist in Section 6.

What you get

Once enabled, the cluster maintains the catalog table at its fixed coordinates (vast-big-catalog-bucket / vast_big_catalog_schema / vast_big_catalog_table) with one row per element in the namespace. No view, no bucket setup, no per-tenant work — tenancy is handled automatically at query time (Section 5, Tenant scoping is automatic).

1. Overview & Prerequisites

Prerequisites

One admin setting underpins everything in this guide:

1. The catalog is enabled and healthy: bigcatalogconfig shows enable: true and state: "working" (Section 0).

With that in place, the catalog is immediately queryable via the VMS REST API (Section 2). The VAST DB SDK reads the same table over the S3/tabular data endpoint; it needs an S3 access key whose user has an s3:Tabular* identity policy, covered in Section 5.

Also worth knowing before your first query:

• Results are as fresh as the last refresh frame. The catalog is rebuilt on the frames.every cadence (15–30 minutes typically), not on every write (Section 0).

• What you see depends on the door you use. The VMS API reads through the management plane and shows the whole cluster — every tenant. The SDK is automatically scoped to the tenant of the querying key (Section 5, Tenant scoping is automatic).

Catalog table coordinates (SDK path only)

The catalog has the same well-known coordinates on every cluster:

The SDK has these names built in — tx.catalog() returns the table handle directly. The VMS API path needs none of this; it addresses the catalog through the management plane.

Choose your access method

• Cluster-wide question, admin hat on? ("what's the biggest data on the cluster?", "how many DB tables exist across tenants?") — the VMS REST API (Section 2, Path A).

• Building a tool, or answering questions inside one tenant's namespace? — the VAST DB SDK (Section 5); the only setup is a key and one identity policy, and tenancy is enforced for you.

• Handing a team self-service access to their data only? Give them SDK access using a key from their tenant — the server handles scoping. Within a single tenant, the query grant is table-level, so finer path-level scoping needs a mediated service, the same pattern as the AuditDB guide's Section 5.

Two audiences, two starting points

• Administrators/platform team: Section 2 to get data flowing today, then Section 5 to set up SDK access for users and tools.

• Analysts/tool builders: start at Section 4 for the schema and recipes, and keep Section 3 (Troubleshooting) nearby — the trailing slash and the refresh cadence cause most first-day confusion.

2. Query the Catalog

The VMS REST API reads the catalog using nothing more than VMS credentials through the management plane as soon as the catalog is enabled — and it sees every tenant's namespace. Start here, and move to the SDK path (Section 5) when you need bulk extracts or want to embed queries in tooling.

Path A: the VMS REST API

The endpoint is bigcatalogconfig/query_data (a POST). It requires two things on every call, and rejects the request with a 400 naming the missing field otherwise:

• path — the subtree to search. It anchors the query and is recursive: path=/ searches the whole namespace, path=/projects/ everything below it.

• filters — at least one server-side row filter (shape below).

Plus the usual: fields (columns to return) and limit.

The quickest way to run one is vastpy-cli:

vastpy-cli --address var202.selab.vastdata.com --user admin --password '********' \
  post bigcatalogconfig/query_data path=/ limit=20 \
  filters='{"element_type":[{"eq":"TABLE"}]}' \
  fields='["parent_path","name","element_type","size","mtime","owner_name"]'

The same call from Python, for anything scripted (vastpy is the same client under the CLI):

import urllib3; urllib3.disable_warnings()
from vastpy import VASTClient

c = VASTClient(address="var202.selab.vastdata.com", user="admin", password="********")

resp = c.bigcatalogconfig.query_data.post(
    path="/",
    limit=50,
    fields=["parent_path", "name", "element_type", "size", "mtime", "owner_name"],
    filters={
        "element_type": [{"eq": "TABLE"}],          # every DB table on the cluster
        # "name":       [{"startswith": "report"}], # startswith also verified on 5.4.4
    },
)
# resp is {"prop_list": [...columns...], "results": [[...row...], ...], "count": N}
cols = resp["prop_list"]
for row in resp["results"]:
    print(dict(zip(cols, row)))

A ready-made, filterable version is in examples/query_vms_catalog.py:

export VMS_ADDRESS=var202.selab.vastdata.com VMS_USER=admin VMS_PASSWORD='********'
python query_vms_catalog.py --path / --type TABLE --top 50
python query_vms_catalog.py --path /home/ --name-prefix report --top 20

Authenticate to VMS

Point vastpy at the management host — the same hostname you open the GUI with, as a bare name (vastpy adds https:// itself):

export VMS_ADDRESS='var202.selab.vastdata.com'
export VMS_USER='admin'
export VMS_PASSWORD='********'
# or, VAST 5.3+:  export VMS_TOKEN='<api-token>'

The S3 data endpoint is a different address and only speaks S3; pointing vastpy at it returns 400 CredentialsNotSupported.

Filter syntax

filters is shaped {"<column>": [{"<operator>": <value>}]}, with the snake_case column names from Section 4. Verified operators on VAST 5.4.4: eq and startswith. Operators the audit endpoint's docs suggest, but this endpoint rejects on 5.4.4 (as 400 with the operator name): contains, gte and other range forms — the set for your cluster is listed at https://<vms>/docs/. For range and aggregate questions, use the SDK path, where the full predicate language is available (Section 4, Recipe queries).

There is no match-all operator: at least one real filter is always required, and an empty-prefix {"startswith": ""} is rejected too (400). When you just want "everything under a path", filter on something broad like element_type.

Three response details worth knowing:

• If you omit fields, results come back with a default column set (parent_path, name, size, mtime, owner/group columns) — pass fields explicitly (POST honors it).

• count can be capped on very large result sets (a broad query on a 5.4.4 lab cluster reports exactly 2097152 = 2²¹). Treat it as "at least this many", not an exact total.

• Timestamps come back as ISO strings (2026-08-06T03:12:10.196574), in UTC.

Sibling endpoints

Same resource, same auth:

Path B: the Web UI

In versions that expose the catalog in VMS, open Database → VAST Catalog in the left menu, then select Open Query Panel. You build conditions per column (the operators offered vary by data type), OR alternative conditions on the same column, AND conditions across columns, and view results in a grid whose displayed columns you can adjust. The official walkthrough is Querying VAST Catalog from the VAST Web UI.

The column names match Section 4, and conditions map onto the filters above. Availability and layout vary by release; this guide's programmatic paths are identical everywhere, so they are what the examples use. If the panel exists on your version, it is the fastest way to confirm the catalog has data.

What this path shows you (and what it doesn't)

The VMS API reads as the management plane: results span every tenant on the cluster. That makes it the right tool for cluster-wide administrative questions — and the wrong tool for handing directly to a tenant's users. For per-tenant access, use the SDK path, which is automatically scoped to the querying key's tenant (Section 5, Tenant scoping is automatic); to give users less than a tenant (a path subtree, say), put a small mediated service in front of either path and inject the caller's allowed prefix into path/filters — the same pattern as the AuditDB guide's Section 5.

When to use the SDK

Move to Section 5 when you need range predicates, large extracts, streaming reads, or queries embedded in tooling. That path reads the table via the S3/tabular data endpoint and requires only an S3 key and a single identity policy.

3. Troubleshooting

Three problems account for almost every support case: the SDK reports a missing bucket, a query runs but returns nothing (or less than you expected), or a query never seems to finish. All have straightforward fixes.

Fastest way to get unblocked: the VMS API path (Section 2) reads the same catalog with just VMS credentials — and sees every tenant. If you're stuck on the SDK, use it to keep working while you fix it.

Error 1: MissingBucket on the SDK path

vastdb.errors.MissingBucket: {'bucket': 'vast-big-catalog-bucket'}

This means the code opened the bucket with tx.bucket("vast-big-catalog-bucket"). That call starts with an S3 HEAD Bucket check, and the catalog is a managed bucket that doesn't answer S3 bucket calls (it isn't backed by a view, so HEAD returns 404, and it's absent from ListBuckets) — even though tabular queries against it work fine. The fix is to open it the way the SDK intends, by well-known name:

with session.transaction() as tx:
    table = tx.catalog()          # instead of tx.bucket("vast-big-catalog-bucket")...

tx.catalog() skips the S3-level check and goes straight to the tabular API (Section 5, The catalog is a managed bucket). If the query then fails with an access error, work down this list:

One thing that looks related but isn't: the catalog never shows up in aws s3 ls or tx.bucket(). That's by design, not a permissions problem — don't go looking for a view or View Policy to fix it.

Error 2: empty or unexpected results

The query runs and returns no errors but returns no rows, or fewer than you expect. Causes, most common first:

2a. Directory listing without the trailing slash

Directory listing is an equality match on parent_path, and the stored values end in /. The slash is not optional:

(_.parent_path == "/projects")    # 0 rows — silently
(_.parent_path == "/projects/")   # the directory's entries

Normalize paths before querying, as examples/first_query.py does.

2b. The path belongs to another tenant

The SDK is automatically scoped to the querying key's tenant. A path that plainly exists — you can see it in the VMS API or the UI — returns zero rows through the SDK when it belongs to a different tenant. That's the feature working as designed (Section 5, Tenant scoping is automatic), and it's the first thing to check when VMS and SDK results disagree. (Live example: on our lab cluster /dataengine/ exists in six tenants but not the default tenant, so a default-tenant key gets an empty listing for it while the VMS API shows it.)

Two related symptoms:

• 403 Forbidden at transaction begin: the key is valid but used against another tenant's VIP pool. Keys work only through their own tenant's VIPs.

• You genuinely need the cross-tenant view: that's the VMS API path (Section 2), not an SDK setting.

2c. The element is newer than the last refresh

The catalog is refreshed on the frames.every cadence (15–30 minutes typically, Section 0) — it is an index, not a live view. A file created two minutes ago is not a bug when it's missing; it appears after the next frame. (Verified live: an object PUT via S3 was queryable over S3 immediately and absent from the catalog in the same minute, on a cluster with a 30 m cadence.) If freshness matters to a report, read the cadence from bigcatalogconfig and say "as of" accordingly.

2d. Wrong element_type spelling

element_type values are UPPERCASE and the set is version-specific. On VAST 5.4.4 and 5.5.0.1: FILE, DIR, SYMLINK, TABLE, SCHEMA, BLOCK_DIR — so a filter like element_type = 'File' or 'file' matches nothing. List what your cluster has (a handful of cheap probes, never a full scan):

$ python3 examples/first_query.py --types
Using the number of endpoints as a heuristic for concurrency.
Using the number of endpoints as a heuristic for concurrency.
Using the number of endpoints as a heuristic for concurrency.
Using the number of endpoints as a heuristic for concurrency.
Using the number of endpoints as a heuristic for concurrency.
Using the number of endpoints as a heuristic for concurrency.
element_type values present on this cluster:
  FILE
  DIR
  SYMLINK
  TABLE
  SCHEMA
  BLOCK_DIR
(candidate list is version-specific; yours may have more — see ../04-schema-and-query-recipes.md)

2e. Filtering on the map columns

user_tags and user_metadata are Arrow maps, and map-key predicates don't push down — (_.user_tags["env"] == "qa") raises NotImplementedError on vastdb 2.x. The pattern that works: push down the counter column, filter the map client-side:

from ibis import _
df = table.select(columns=["parent_path", "name", "user_tags"],
                  predicate=(_.user_tags_count > 0),        # pushes down
                  limit_rows=100_000).read_all().to_pandas()
df["tags"] = df["user_tags"].map(dict)
hits = df[df["tags"].map(lambda t: t.get("env") == "qa")]   # client-side

(Section 4, Recipe 5 shows the full version.)

On the VMS path, the analogous surprise is operators: only eq and startswith are confirmed on 5.4.4; anything else returns 400 naming the operator (Section 2, Filter syntax).

Error 3: the query never finishes

The catalog holds one row per element on the cluster — billions on a big system (our 5.4.4 lab: ~1.0 billion elements, 3.3 billion physical rows). Every select() is a server-side scan with predicate pushdown, and two things decide how long it runs (timings below measured live on that lab):

• Queries return early once limit_rows is satisfied. Matches stream back as the scan finds them: a 1000-entry directory listing with limit_rows=1000 returned in ~3 s; the same 5-entry directory took ~1.7 s with limit_rows=5.

• When matches are fewer than limit_rows, the scan must sweep the whole catalog to be sure it's done — ~15 s on our 3.3 B-row lab (~9 s with num_splits=64), regardless of how selective the predicate is. Listing a 5-entry directory with limit_rows=1000 costs the full sweep. That's the per-query floor, and it grows with catalog size.

• Big result sets add transfer time on top. Streaming a 200 k-row subtree took ~15 s; an unbounded read_all() over millions of rows runs for minutes and looks exactly like a hang.

The habits that follow:

# 1. Always pass limit_rows — it caps the fetch AND lets the scan stop early
reader = table.select(columns=cols, predicate=pred, limit_rows=10_000)

# 2. Keep limit_rows close to what you'll actually consume (page in modest
#    chunks), and anchor subtree predicates as deep as the question allows —
#    both raise the odds the scan can stop before sweeping everything

For genuinely large scans — space rollups, subtree inventories — iterate the reader in batches instead of read_all(), and raise QueryConfig.num_splits so the sweep fans out (Section 5, Reading at scale).

A related hang happens before any query runs, when the endpoint is wrong. The failure mode depends on what's at the address: a host that answers but isn't a VAST DB endpoint (the VMS address, say) fails immediately (UnsupportedServer), but an unreachable host hits vastdb.connect()'s default retry loop — still retrying after 2½ minutes in our test. For interactive tools, fail fast instead (verified: 5 s):

from vastdb.config import BackoffConfig
session = vastdb.connect(endpoint=EP, access=AK, secret=SK,
                         timeout=5, backoff_config=BackoffConfig(max_tries=1))

All three example scripts in examples/ do exactly this (their --timeout flag).

Quick triage flow

MissingBucket from the SDK?              → Error 1: open by name — tx.catalog(), not tx.bucket()
Directory listing returns 0 rows?        → 2a: trailing slash on parent_path
VMS shows the path, SDK returns nothing? → 2b: it's another tenant's namespace (by design)
403 at transaction begin?                → 2b: key used against another tenant's VIP pool
A just-created file is missing?          → 2c: wait for the next refresh frame (Section 0 cadence)
element_type filter matches nothing?     → 2d: UPPERCASE, version-specific — run --types
Predicate raises NotImplementedError?    → 2e: map columns — push down user_tags_count instead
Query hangs / runs forever?              → Error 3: unbounded fetch — add limit_rows, batch it
Every query takes ~15s even for 5 rows?  → Error 3: normal — low-match queries sweep the catalog
Connect itself hangs?                    → Error 3: unreachable endpoint — use the DB data VIP; cap timeout
Nothing via the VMS path either?         → catalog isn't enabled/working (Section 0)

4. Schema & Query Recipes

Recipes are shown with the VAST DB SDK (ibis/pandas). Equality and prefix filters map directly onto the VMS REST API (fields + filters, Section 2, Filter syntax); range and aggregate queries are SDK-only there.

The schema

One row per element in the namespace. The full column list on VAST 5.4.4 (32 columns — also retrievable without the SDK via GET bigcatalogconfig/columns/; the official schema reference is in the VAST Catalog Overview):

Identity: what and where

Size and links

Ownership and permissions

Times (all timestamp[ns], UTC)

S3 object metadata

Devices

Recipe queries (VAST DB SDK)

Each assumes an open table handle:

import vastdb
from ibis import _
with vastdb.connect(endpoint=<endpoint>, access=<access_key>, secret=<secret_key>).transaction() as tx:
    table = tx.catalog()   # the VAST Catalog, by its built-in well-known name
    ...                    # recipes below run inside this block

Two habits to keep: always pass limit_rows (it caps the fetch and lets the scan stop early), and keep it close to what you'll actually consume — a query whose matches are fewer than its limit_rows sweeps the whole catalog before returning (Section 3, Error 3).

For the most common reports, examples/catalog_report.py is ready to run:

python catalog_report.py --largest --under /projects/
python catalog_report.py --owner root --changed-since '2026-08-01'
python catalog_report.py --ext parquet --min-size 1G
python catalog_report.py --tagged

Recipe 1: List a directory

Equality on parent_path, trailing slash mandatory (Section 3, 2a):

df = table.select(
    columns=["name", "element_type", "size", "used", "mtime", "owner_name"],
    predicate=(_.parent_path == "/projects/models/"),
    limit_rows=100_000,
).read_all().to_pandas()

Filter to subdirectories only with & (_.element_type == "DIR") — that's how a tree browser walks the namespace one query at a time. Expect seconds, not milliseconds: a listing that doesn't fill its limit_rows sweeps the catalog before it can declare itself done (~15 s on a 3 B-row lab; Section 3, Error 3), which is why browsing tools cache listings.

Recipe 2: Largest files under a subtree

startswith scans the subtree, so give the scan a selective size floor and let the sort happen client-side:

pred = (_.element_type == "FILE") \
       & (_.parent_path.startswith("/projects/")) \
       & (_.size >= 1 * 2**30)                      # ≥ 1 GiB
df = table.select(columns=["parent_path", "name", "size", "owner_name"],
                  predicate=pred, limit_rows=100_000).read_all().to_pandas()
top = df.sort_values("size", ascending=False).head(100)

Caveat: if the scan hits limit_rows, you sorted a sample, not the true top-N — raise the size floor until the match set fits under the cap.

Recipe 3: Everything that changed since a point in time

import ibis
from datetime import datetime
pred = (_.mtime >= ibis.literal(datetime(2026, 8, 1), type="timestamp")) \
       & (_.parent_path.startswith("/projects/"))
df = table.select(columns=["parent_path", "name", "element_type", "mtime", "owner_name"],
                  predicate=pred, limit_rows=100_000).read_all().to_pandas()

Times are UTC. creation_time, ctime, and atime take the same shape — "files nobody has read in a year" is this recipe with _.atime <=.

Recipe 4: One owner's data (or one extension)

pred = (_.element_type == "FILE") & (_.owner_name == "jsmith")       # or _.uid == 1000
pred = pred & (_.extension == "parquet")                              # optional
df = table.select(columns=["parent_path", "name", "size", "mtime"],
                  predicate=pred, limit_rows=100_000).read_all().to_pandas()
print(df["size"].sum(), "bytes in", len(df), "files")

Recipe 5: Everything with S3 user tags

Map columns don't push down; user_tags_count does (Section 3, 2e):

df = table.select(columns=["parent_path", "name", "user_tags", "user_metadata"],
                  predicate=(_.user_tags_count > 0),
                  limit_rows=100_000).read_all().to_pandas()
df["tags"] = df["user_tags"].map(dict)
qa = df[df["tags"].map(lambda t: t.get("env") == "qa")]   # client-side key/value filter

Recipe 6: Count elements by type (a full-scan report, done deliberately)

Exact cluster-wide counts read one column across the entire catalog — minutes on a big cluster, so stream it instead of read_all() and know you asked for it:

from collections import Counter
counts = Counter()
reader = table.select(columns=["element_type"])       # no limit: deliberate full scan
for batch in reader:
    counts.update(batch.column("element_type").to_pylist())
print(dict(counts))

Scope it to a subtree with a parent_path.startswith predicate to make the transfer proportionally cheaper. (For "which types exist at all", skip the scan — that's first_query.py --types, six limit_rows=1 probes. For a plain total element count, skip it too: GET bigcatalogconfig/stats returns count_rows (≈1.0 B on our lab), and the SDK's table.get_stats() returns the physical row count — larger, ~3.3 B on the same lab, because the table retains multiple catalog frames.)

5. Advanced: SDK Access & Tenancy

The VAST DB SDK reads the catalog the same way it reads any VAST Database table: over S3, with an access key. Use it for anything programmatic — reports, pipelines, tools. This section walks through the setup, then the two things that make the catalog different at scale: automatic tenant scoping, and reading efficiently from a table with a row for every element on the cluster.

Connect and query

Three steps: install, point at the data endpoint, open the catalog by name.

pip install vastdb pyarrow pandas ibis-framework
export VASTDB_ENDPOINT='http://<cluster-db-data-vip>'   # S3/tabular data VIP, not the VMS address
export VASTDB_ACCESS_KEY='<access-key>'
export VASTDB_SECRET_KEY='<secret-key>'

import os, vastdb
from ibis import _

session = vastdb.connect(endpoint=os.environ["VASTDB_ENDPOINT"],
                         access=os.environ["VASTDB_ACCESS_KEY"],
                         secret=os.environ["VASTDB_SECRET_KEY"])
with session.transaction() as tx:
    table = tx.catalog()                  # the VAST Catalog, by its built-in well-known name
    batch = table.select(
        columns=["name", "element_type", "size", "mtime", "owner_name"],
        predicate=(_.parent_path == "/projects/"),
        limit_rows=100,
    ).read_all()
    print(batch.to_pandas())

That's the whole pattern. tx.catalog() returns a regular table handle, so everything in Section 4 applies as-is. The starter examples/first_query.py wraps this with listing, --types, and --schema; examples/catalog_report.py builds it out into the common reports.

One habit for command-line tools: pass timeout= and backoff_config=BackoffConfig(max_tries=1) (from vastdb.config) to vastdb.connect() so an unreachable endpoint fails in seconds (measured: 5 s, vs. still-retrying at 2½ minutes with the defaults; Section 3, Error 3).

The catalog is a managed bucket

The catalog lives at fixed coordinates on every cluster — vast-big-catalog-bucket / vast_big_catalog_schema / vast_big_catalog_table — and the SDK knows them (tx.catalog(), just like tx.audit_log() for the AuditDB). Because the bucket is managed by the cluster rather than backed by a view you created, it behaves differently from your own DB buckets in one visible way: it doesn't answer S3 bucket calls. It won't appear in ListBuckets (aws s3 ls), and tx.bucket("vast-big-catalog-bucket") fails with MissingBucket because that call begins with an S3 HEAD Bucket check. This is normal — open it by name and query it (Section 3, Error 1).

On an SDK too old to have tx.catalog(), or in a tool that keeps the coordinates configurable, construct the handle by name explicitly — it's the same open-by-name operation:

import vastdb.bucket, vastdb.schema
b = vastdb.bucket.Bucket("vast-big-catalog-bucket", tx)
table = vastdb.schema.Schema("vast_big_catalog_schema", b).table("vast_big_catalog_table")

Grant access: the identity policy

The querying user needs an S3 access key and one read-only identity policy with the tabular actions. That's the entire permission model for the SDK path — the catalog bucket has no View Policy to join, because it isn't backed by a view, and there is no per-tenant catalog setup.

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "ListTabular", "Effect": "Allow",
      "Action": "s3:TabularList*",
      "Resource": ["vast-big-catalog-bucket", "vast-big-catalog-bucket/vast_big_catalog_schema/*"] },
    { "Sid": "QueryTabular", "Effect": "Allow",
      "Action": "s3:TabularQueryData",
      "Resource": ["vast-big-catalog-bucket/vast_big_catalog_schema/*"] }
  ]
}

• s3:TabularList* lets the SDK resolve the schema and table; s3:TabularQueryData performs the read. Both are needed for a working read-only client.

• Resource forms (from VAST's Managing Permissions for Accessing VAST Tabular Databases): "<bucket>" for bucket-level ops, "<bucket>/<schema>/*" for schema-level, "<bucket>/<schema>/<table>" for table-level. The arn:aws:s3:::<bucket>/... form is also accepted on some versions. A broad Resource: "*" also matches the managed catalog bucket — verified live on VAST 5.5.0.1 with a from-scratch user whose only grant was a tabular-actions policy on "*".

Setup order, all with vastpy (see the cheat-sheet in Section 6): create the user, create their access key, create the identity policy, attach it to the user.

Tenant scoping is automatic

The catalog is scoped, server-side, to the tenant of the S3 identity you query with. A tenant's key sees exactly its own namespace — its paths, its rows, nothing else. There is no tenant parameter to pass and no way to opt out from the SDK side. Verified live (VAST 5.4.4): a default-tenant key listing / sees only default-tenant paths; a second tenant's subtrees simply aren't there; a key used against another tenant's VIP pool gets 403 Forbidden at transaction begin (the tenant is bound to the VIP pool, so use the tenant's own VIPs).

Practical consequences:

• Giving a team access to their namespace is zero extra work: create a user in their tenant, attach the identity policy above, hand out a key for their tenant's VIP pool. The server does the scoping.

• Cross-tenant reporting is not an SDK capability — that's the VMS API path (Section 2), which reads as the management plane and sees everything.

• Don't try to reimplement tenancy with the tenant_id column. It holds internal IDs that don't match VMS tenant IDs (and your own rows can show -1). The scoping already happened before your predicate ran.

• If you also use the AuditDB, note the inversion: audit rows are cluster-wide to any key that can read the table (you filter on its tenant name column yourself), while the catalog is auto-scoped. Same access pattern, opposite tenancy defaults.

Scoping below a tenant

Within one tenant, the s3:TabularQueryData grant is table-level — all-or-nothing. To give a user only a path subtree, put a small mediated query service in front: it holds the credential, maps each authenticated caller to their allowed prefixes, and injects the prefix into the predicate (_.parent_path.startswith(allowed)) or the VMS call's path/filters. The pattern (and its design notes) is the AuditDB guide's Section 5 — it applies here unchanged.

Out of scope: Trino and Row and Column Security

External Trino query engines can also query the catalog, and VAST Database Row and Column Security can make the database itself filter rows and columns per user — but RCS is enforced only by Trino engines with end-user impersonation, so both are out of scope for this guide. If you later need a hard, database-enforced boundary below the tenant level, start with VAST's VAST Database Row and Column Security doc.

Reading at scale

Two SDK features matter once queries leave "one directory" territory:

Stream, don't buffer. select() returns a RecordBatch reader. read_all() buffers everything; iterating processes batches as they arrive, in bounded memory:

reader = table.select(columns=["parent_path", "size", "used"],
                      predicate=(_.element_type == "FILE")
                                & (_.parent_path.startswith("/projects/")))
total = 0
for batch in reader:                      # arrives while the scan still runs
    total += sum(batch.column("used").to_pylist())

Fan the scan out. QueryConfig controls parallelism — num_splits (how the table scan is partitioned) and num_sub_splits per split. Measured on a 3.3 B-row lab catalog: num_splits=64 cut a full-sweep query from ~15 s to ~9 s and a 200 k-row subtree stream from ~15 s to ~10 s; production tools also spread requests across all the data VIPs (QueryConfig(data_endpoints=[...])):

from vastdb.config import QueryConfig
config = QueryConfig(num_splits=64, num_sub_splits=4)
reader = table.select(columns=cols, predicate=pred, config=config)

Know the size without scanning. table.get_stats() returns the table's physical row count and byte size in one cheap call (num_rows=3_300_583_568 on our lab — note this is physical rows including retained catalog frames, larger than the element count that GET bigcatalogconfig/stats reports).

What you can build on this: Catwalk

A worked example of everything in this section is Catwalk, an interactive web file browser for the VAST Catalog: pick a view, browse the tree, page through directories with millions of entries, and get per-directory space rollups aggregated over all descendants — all from catalog queries, never walking the filesystem. Internally it is exactly this guide's patterns: tx.catalog(), equality-on-parent_path listings (cached, because even a small listing costs a catalog sweep — Section 3, Error 3), streamed startswith scans with num_splits fan-out for rollups, and per-tenant deployment by credential (run it with a tenant's key and it browses that tenant's namespace — no tenant option needed, see above). It also ships a mock mode (CATWALK_MOCK=1) that serves a synthetic namespace, handy for demos without a cluster.

6. Appendix: CLI, Config Links & Verification

Environment variables (SDK / companion scripts)

The examples/ scripts read these:

# SDK path (first_query.py, catalog_report.py)
export VASTDB_ENDPOINT='http://<cluster-vip>'   # VAST DB data endpoint (not VMS)
export VASTDB_ACCESS_KEY='<access-key>'
export VASTDB_SECRET_KEY='<secret-key>'

# VMS path (query_vms_catalog.py)
export VMS_ADDRESS='<vms-host>'                 # bare hostname; vastpy adds https://
export VMS_USER='admin'
export VMS_PASSWORD='********'                  # or VMS_TOKEN (VAST 5.3+)

The catalog bucket/schema/table names are fixed (vast-big-catalog-bucket / vast_big_catalog_schema / vast_big_catalog_table) and built into the SDK, so they never need configuring.

vastpy cheat-sheet

vastpy is a thin VMS REST client (SDK and vastpy-cli). It's a schema-less passthrough: vastpy-cli <get|post|patch|delete> <endpoint> key=value .... Values that look like JSON are parsed as JSON, and -i file.json supplies a POST/PATCH body from a file.

# --- Auth (env or per-call flags) ---
export VMS_ADDRESS=vms.example.com
export VMS_USER=admin
export VMS_PASSWORD='********'

# --- Catalog config (admin; full enable procedure in Section 0) ---
vastpy-cli get bigcatalogconfig                       # enabled? state? refresh cadence?
vastpy-cli patch bigcatalogconfig/1 enable=true \
  frames='[{"every":"15m","every_value":"15m","every_unit":"m",
            "start-at":"<keep the value from the GET>",
            "keep-local":"2H","local-period":"H",
            "keep-remote":"15m","remote-period":"m"}]'

# --- Query the catalog via the VMS API ---
# POST; both `path` (subtree anchor, recursive) and `filters` are required.
vastpy-cli post bigcatalogconfig/query_data path=/ limit=20 \
  filters='{"element_type":[{"eq":"TABLE"}]}' \
  fields='["parent_path","name","element_type","size","mtime","owner_name"]'

# --- Schema without SDK access ---
vastpy-cli get bigcatalogconfig/columns

# --- Sibling endpoints (see Section 2) ---
vastpy-cli get bigcatalogconfig/stats
vastpy-cli get bigcatalogindexedcolumns

# --- Users, keys & identity policies (SDK-path setup; see Section 5) ---
vastpy-cli post users name='cataloguser' user_type='LOCAL'
vastpy-cli post users/<user_id>/access_keys           # or .../s3_access_keys on some versions
vastpy-cli post s3policies name='catalog-readonly' policy="$(cat policy.json)" tenant_id=1
vastpy-cli patch s3policies/<policy_id> users='["cataloguser"]'   # attach (version-dependent)

A note on TLS: vastpy doesn't verify certificates unless you pass --cert-file / cert_file, and the same goes for vastdb.connect(..., ssl_verify=False). Fine for labs; set a CA bundle in production.

VAST DB tabular permissions

Tabular operations are granted by an identity policy with s3:Tabular* actions, scoped by Resource: "<bucket>" (bucket-level, e.g. list schemas), "<bucket>/<schema>/*" (schema-level), "<bucket>/<schema>/<table>" (table-level).

For the catalog (a managed bucket, like the AuditDB), the identity policy is the whole model — there is no view or View Policy involved, and tenancy is enforced server-side at query time (Section 5, Tenant scoping is automatic). For DB buckets you create yourself (S3-enabled DATABASE views), the user must additionally be in the view's View Policy with bucket-listing permission for the bucket to be visible.

Common tabular actions (identity-policy Action values, all s3:-prefixed):

Official VAST references

This guide gets you started; these are the reference docs:

• VAST Catalog Overview — concepts, the official schema reference, and the supported query tools.

• Configuring VAST Catalog — the Web UI enablement procedure, snapshot schedules and retention, user-defined attribute indexing (S3 tags/metadata), and monitoring indexing progress.

• Querying VAST Catalog from the VAST Web UI — the Query Panel walkthrough (Database → VAST Catalog → Open Query Panel).

• Managing Permissions for Accessing VAST Tabular Databases (support KB) — the identity-policy model and Resource forms used in Section 5.

• VAST Database Row and Column Security (support KB) — only if you later need database-enforced scoping (Trino-only; out of scope here).

• VAST's data-platform-field-docs — runnable catalog notebooks, including the Python SDK examples this guide's recipes align with.

Verification checklist

Facts this guide asserts, and where they were last confirmed. A full skeptical re-verification of every claim ran 2026-08-06 against var202 (VAST 5.4.4.0, vastdb 2.0.14); rows below are from that pass unless noted. The examples were additionally re-run the same day against var204 (VAST 5.5.0.1). Re-verify the version-specific rows on a new VAST release.

Enablement & VMS API

SDK path

Performance (measured — the Section 3 Error 3 cost model)