How To Audit DB Query Guide

Enable Audit Logging to the Database

This page turns on protocol auditing for all NFS and S3 views on a VAST cluster and stores the records in the VAST Database (the AuditDB), from which the rest of this guide can query them. It takes two settings and about five minutes.

The two settings:

1. Save audit logs to VAST DB: Routes audit records into a database table.

2. Global Baseline Audit Settings: Choose which protocols and operations get recorded. Whatever you enable here applies to every view on the cluster, regardless of view policy. That is why covering all NFS and S3 views takes no per-view or per-policy work. (View policies can add auditing selectively on top of the baseline; you only need that when you want some views audited more than others.)

Enable it in the Web UI

1. Log in to VMS and go to Settings → Auditing.

2. Leave the General settings at their defaults. The audit directory name and the
   Read-access Users/Groups fields are part of the JSON file export, which is a separate path from the database.

3. Under Define retention period, set the length of time audit data is retained. The default is a
   manually defined period of one hour, which is short; pick a window that matches how far back you need to look.

4. Under Auditing, turn on Save audit logs to VAST DB. This is the switch that creates the AuditDB table. (Saving audit logs to a file (JSON format) is independent; you can enable both.)

5. Under Global Baseline Audit Settings:

   1. Toggle Enable on.

   2. Under Select protocols to assign operations, select the protocols your views serve. For all NFS and S3 views, select NFSv3, NFSv4.1, and S3. (The full list on 5.5 is NFSv3, NFSv4.1, SMB, S3, and NDB, where NDB is VAST Database traffic.)

   3. Under Operations to audit, choose the categories to record:

      Category	Records
      Create/Delete Files/Directories/Objects	Creates and deletes
      Modify Data	Writes, including size changes
      Modify Metadata	Permission, ownership, attribute changes
      Read Data	Reads and downloads
      Read Metadata	Stats, lists, lookups
      Session create/close	Kerberos session open/close

      A good starting set is Create/Delete, Modify Data, and Modify Metadata. Add the Read categories deliberately: on a busy cluster, reads dominate traffic, so they multiply the row volume.

   4. Under the audit record options, leave Log Full Path on (the default; each record carries the full Element Store path, at a small performance cost) and turn on Log Username so records carry the resolved username, not just the UID.

6. Click Save.

That's it. Rows begin to appear as audited operations occur.

Or enable it with the VMS REST API

The same settings live at the clusters/<id>/auditing endpoint, so this script cleanly with vastpy:

pip install vastpy
export VMS_ADDRESS=<vms-host>        # management host, bare hostname
export VMS_USER=admin
export VMS_PASSWORD='********'

vastpy-cli get clusters              # find your cluster id (usually 1)
vastpy-cli get clusters/1/auditing   # current settings

Write the config you want as audit-config.json. Field names below are as returned by the GET on VAST 5.5.0.1; run the GET first and mirror the names your version returns (modify_data_md and read_data_md are the metadata categories):

{
  "enable_vast_db_audit": true,
  "protocols": ["NFSv3", "NFSv4.1", "S3"],
  "protocols_audit": {
    "create_delete_files_dirs_objects": true,
    "modify_data": true,
    "modify_data_md": true,
    "read_data": true,
    "read_data_md": true,
    "log_full_path": true,
    "log_username": true
  }
}

Apply it, then confirm:

vastpy-cli patch clusters/1/auditing --file-input audit-config.json
vastpy-cli get   clusters/1/auditing

The confirming GET should show enable_vast_db_audit: True and, once the table exists, vast_audit_log_state: CREATED.

VAST CLI works too if you prefer it: cluster show --audit displays the current settings and cluster modify --audit-protocols ... --audit-operations ... --enable-audit-settingschanges them; see VAST's Configuring Global Auditing Settings KB page.

Confirm it's working

Generate some traffic first (write a file over NFS, or put an object into an S3 bucket), then read it back either way:

• Web UI: go to DataBase → VAST Audit Log. Your operations appear as rows. Scripted: python examples/query_vms_api.py --bucket <bucket> --top 10 (§2 covers this path).

From here, the audit table is fully queryable through the Web UI, the VMS REST API, and the VAST DB SDK. The SDK path just needs an S3 key and an identity policy, covered in §5.

1. Overview & Prerequisites

Prerequisites

Everything in this guide builds on three admin settings. Confirm them once, and you're set (if any are off, §0 walks through enabling them):

1. Protocol auditing is enabled on the cluster.

2. Audit logs are saved to VAST DB. In VMS: Settings → Auditing → Save audit logs to VAST DB. Both Save to VAST DB" and "Save to file (JSON)" can be on at once; this guide uses the VAST DB destination. (To read the JSON files over NFS, S3, or SMB instead, see VAST's Accessing Audit Log JSON Files via NFS, S3, and SMB.)

3. The operations you care about are audited. Auditing is opt-in per category (Create/Delete, Modify Data, Modify Metadata, Read Data, Read Metadata, Sessions). Rows appear only for enabled categories.

With those in place, the audit table is readable through the Web UI and the VMS REST API immediately (§2). The VAST DB SDK reads the same table over the S3/tabular data endpoint; it needs an S3 access key whose user has an s3:Tabular* identity policy, covered in §5.

The step-by-step enable procedure is in §0; the authoritative references are VAST's Configuring Global Auditing Settings and Configuring Auditing with View Policies KB pages (§6).

Audit table coordinates (SDK path only)

The AuditDB has the same well-known coordinates on every cluster:

The SDK has these names built in — tx.audit_log() returns the table handle directly. The Web UI and VMS API paths do not need this; they access the audit log through the management plane.

Choose your access method

• Just need to look, or confirm data exists? Use the Web UI Query Panel (§2A). No keys, no policy, nothing to install.

• Automating, or building a service that hands out per-user scoped results? Use the VMS REST API (§2B). One privileged credential: your service applies any per-user path scoping (§5).

• Bulk extracts, pipelines, embedding queries in tooling? Use the VAST DB SDK (§5); the only setup is a key and one identity policy.

One design point worth knowing up front: the query grant (s3:TabularQueryData) operates at the table level, so per-user path scoping is done in a small query service rather than in the policy itself. Section 5 shows the pattern.

Two audiences, two starting points

• Administrators/platform team: §2 to get data flowing today, then §5 to set up SDK access and scoped access.

• Analysts: start at §2 and keep §3 Troubleshooting nearby.

2. Query the AuditDB

Two paths read audit records with nothing more than VMS credentials: the Web UI Query Panel and the VMS REST API. Both read the audit table through the management plane, so they work as soon as auditing is enabled. Start here and move to the SDK path when you need bulk extracts or want to embed queries in tooling.

Path A: the Web UI Query Panel

The fastest way to look at audit data. Nothing to install.

1. In VMS, go to DataBase → VAST Audit Log. The grid shows recent audit records.

2. Use the Protocol drop-down (upper-right) to filter by NFS, SMB, S3, and so on.

3. Click the column-selector icon to choose which fields are shown.

4. Click Open Query Panel to build a query:

   • In the Active Query tab, pick a Column name, choose a Select operator (the operators offered depend on the column's type, e.g., is exactly, is null), and enter a value.

   • Add more clauses with & (AND), or alternatives with OR. Click Add to add each condition to the list.

   • Click Execute Query. Results appear in the grid.

5. Use the download icon to export the currently displayed rows to CSV.

Good for spot-checks, confirming data exists, and giving a non-technical user a way to look without credentials or code.

Path B: the VMS REST API

For anything programmatic, query the same table through the VMS management API with vastpy. This is the path to automate on and the natural base for a gated internal service.

Authenticate to VMS

Point vastpy at the management host, the same hostname you open the GUI with, as a bare name (vastpy adds https:// itself):

export VMS_ADDRESS='var204.selab.vastdata.com'
export VMS_USER='admin'
export VMS_PASSWORD='********'
# or, VAST 5.3+:  export VMS_TOKEN='<api-token>'

The S3 data endpoint is a different address and only speaks S3; pointing vastpy at it returns 400 CredentialsNotSupported. Management calls always go to the management host.

Pull audit rows

The endpoint is vastauditlog/query_data (a POST). Pass fields (columns to return), filters (server-side row filters), and limit:

import urllib3; urllib3.disable_warnings()
from vastpy import VASTClient

c = VASTClient(address="var204.selab.vastdata.com", user="admin", password="********")

resp = c.vastauditlog.query_data.post(
    limit=50,
    fields=["time", "login_name", "protocol", "rpc_type", "view_path", "num_bytes"],
    filters={
        "s3_bucket_name": [{"eq": "mputest"}],   # server-side filter (operator: eq)
        # "view_path":    [{"startswith": "/bgolliher/"}],   # verify operator name on your version
    },
)
# resp is {"prop_list": [...columns...], "results": [[...row...], ...], "count": N}
cols = resp["prop_list"]
for row in resp["results"]:
    print(dict(zip(cols, row)))

The same call from the CLI (which renders the result as a table):

vastpy-cli --address var204.selab.vastdata.com --user admin --password '********' \
  post vastauditlog/query_data limit=20 \
  filters='{"s3_bucket_name":[{"eq":"mputest"}]}' \
  fields='["time","login_name","protocol","rpc_type","view_path","num_bytes"]'

A ready-made, filterable version is in examples/query_vms_api.py:

export VMS_ADDRESS=var204.selab.vastdata.com VMS_USER=admin VMS_PASSWORD='********'
python query_vms_api.py --bucket mputest --top 10
python query_vms_api.py --path /bgolliher/ --top 20

Filter syntax

Every call includes a filters field shaped as {"<field>": [{"<operator>": <value>}]}. The endpoint requires it; a call without one returns 400 filters - field required. The always-available operator is eq; others such as startswith and range operators vary by version, and the full set for your cluster is listed at https://<vms>/docs/. Field names are the snake_case column names from §4; view_path is handy for path filtering because it's a flat string (/bgolliher/mputest).

Scoping results by path

Both paths filter by path: a view_path filter in Path B, or a Query Panel condition in Path A. That filtering is applied by the caller, so it's a convenience rather than a per-user restriction.

• For a convenience filter ("just show me /bgolliher/*"), use the view_path filter above.

• To restrict each requester to their own prefix, wrap Path B in a thin service that injects the caller's allowed prefix into filters and holds the VMS credential itself. Your code is the enforcement point, and this is the recommended way to give UserA /bgolliher/* and nobody else's paths. See §5.

When to use the SDK

Move to §5 when you need the VAST DB SDK for large programmatic extracts or to embed queries in tooling. That path reads the table over the S3/tabular data endpoint and needs only an S3 key and one identity policy, covered there.

3. Troubleshooting

Three problems account for almost every support case: the SDK reports a missing bucket, a query runs but returns nothing, or a query never seems to finish. All have straightforward fixes.

Fastest way to get unblocked: the Web UI and VMS API paths read the same audit data with just VMS credentials. If you're stuck on the SDK, use those to keep working while you fix it.

Error 1: MissingBucket on the SDK path

vastdb.errors.MissingBucket: {'bucket': 'vast-audit-log-bucket'}

This means the code opened the bucket with tx.bucket("vast-audit-log-bucket"). That call starts with an S3 HEAD Bucket check, and the AuditDB is a managed bucket that doesn't answer S3 bucket calls (it isn't backed by a view, so HEAD returns 404, and it's absent from ListBuckets) — even though tabular queries against it work fine. The fix is to open it the way the SDK intends, by a well-known name:

with session.transaction() as tx:
    table = tx.audit_log()          # instead of tx.bucket("vast-audit-log-bucket")...

tx.audit_log() skips the S3-level check and goes straight to the tabular API (§5). If the query then fails with an access error, work down this list:

A few things that look related but aren't:

• The AuditDB never shows up in aws s3 ls or tx.bucket(). That's by design, not a permissions problem — don't go looking for a view or View Policy to fix it.

• Read-access Users / Groups (in the global auditing settings) control access to the audit JSON files, not the database. Per VAST's Accessing Audit Log JSON Files via NFS, S3, and SMB, "access to the Audit Directory and its contents is controlled by the Audit settings." Being on that list is neither necessary nor sufficient for DB access.

• An S3 bucket named audit is the JSON file bucket (it holds audit_env_N/ folders of audit_log_* blobs, owned by root, read-only). It's a different path with a different access model; the database is reached only through the tabular API.

Error 2: empty or unexpected results

The query runs without errors but returns no rows or gives different results than you expect. Causes, most common first:

2a. Wrong rpc_type or protocol value

rpc_type values are case-sensitive, and the exact spelling is cluster/version-specific. The styles, by protocol:

On VAST 5.4/5.5, S3 writes are stored as PUT_OBJECT, COPY_OBJECT, and COMPLETE_MULTIPART_UPLOAD, not the CamelCase PutObject the operation-list doc implies. The doc names the operations; the audit table stores VAST's internal rpc_type spelling, which can change between versions. So a filter like rpc_type = 'PutObject' can return nothing on one cluster and everything on another.

The fix is simple: list the real values from your own data before writing a filter.

python examples/first_query.py --distinct-ops

Or directly with the SDK:

# SDK: list the distinct rpc_type values in a recent sample
import vastdb
with vastdb.connect(endpoint=EP, access=AK, secret=SK).transaction() as tx:
    t = tx.audit_log()
    df = t.select(columns=["protocol", "rpc_type"], limit_rows=100_000).read_all().to_pandas()
    print(df.value_counts(["protocol", "rpc_type"]))

One version note: before VAST 5.4.1, S3 DeleteByLifecycleRule operations are recorded as DeleteObject. Filter accordingly on older clusters.

The protocol column has the same property: the stored spelling can vary by release (NFSv3 vs NFS3 vs NFSV3), and NFSv4.0 and 4.1 share a single NFSv4 value. Filter with isin over the known variants rather than a single equality — examples/audit_timeslice.py shows the pattern:

predicate &= _.protocol.isin(["NFSv3", "NFS3", "NFSV3"])   # matches every spelling

2b. Time-range filters

The time column is a timestamp(9). If your window is wrong (timezone, UTC vs local, or a range in the future), you get zero rows. Widen the window to sanity-check, then narrow.

2c. The path filter and the nested struct

path is a struct; the string lives at path.path. Filtering on path directly won't match the way you expect. See §4.

The audit stream also contains internal noise you usually want to exclude: rows where path.path is null or starts with (null), rows under /.vast_audit_dir/, and the audit service account. In the SDK, exclude the service account in the predicate, and the path noise client-side — negated prefix filters (~startswith) don't push down (they raise NotImplementedError on vastdb 2.x; positive startswith filters push down fine):

from ibis import _
predicate = predicate & (_.login_name != "audit")      # pushes down
df = table.select(columns=cols, predicate=predicate, limit_rows=cap).read_all().to_pandas()

df["path"] = df["path"].map(lambda p: p.get("path", "") if isinstance(p, dict) else p or "")
df = df[~df["path"].str.startswith(("(null)", "/.vast_audit_dir/"))]   # client-side

2d. A policy that seems to scope by path only sometimes

If a per-user policy appears to restrict paths inconsistently, it is because it expects row-level scoping from a table-level grant. s3:TabularQueryData grants the whole table, and narrowing the policy Resource to imply a path prefix doesn't filter rows. To scope what a user sees by path, enforce it in a mediated service that adds the caller's allowed prefix to the query filter (§5).

Error 3: the query never finishes

An SDK query with no row cap fetches every matching row before returning, and a busy cluster writes millions of audit rows per hour, so an open-ended table.select(...).read_all() can run for a very long time while looking "hung". Filter pushdown itself is fast — a selective predicate plus a row cap return in well under a second even on a large table.

Two habits prevent it:

# 1. Always pass limit_rows unless you truly want the full result set
reader = table.select(columns=cols, predicate=pred, limit_rows=10_000)

# 2. Keep time windows tight, and remember rows are not returned newest-first:
#    narrow with the predicate, then sort what comes back

For genuinely large extracts, iterate the reader in batches instead of read_all(), or narrow the window and paginate by time.

A related hang happens before any query runs: pointed at the wrong endpoint (the VMS address or a host that isn't reachable), vastdb.connect() has no request timeout and retries for about a minute by default. For interactive tools, fail fast instead:

from vastdb.config import BackoffConfig
session = vastdb.connect(endpoint=EP, access=AK, secret=SK,
                         timeout=5, backoff_config=BackoffConfig(max_tries=1))

examples/audit_timeslice.py does exactly this (its --timeout flag), so a mistyped endpoint fails in seconds with a pointer to the fix.

Quick triage flow

MissingBucket from the SDK?             → Error 1: open by name — tx.audit_log(), not tx.bucket()
Query runs but returns 0 rows?          → 2a rpc_type value, then 2b time, then 2c path/noise
Data shows in the UI but not my query?  → connection/filters (2a-2c), not auditing
Trying to scope a policy by path?       → 2d: use a mediated service (§5)
Query hangs / runs forever?             → Error 3: add limit_rows, tighten the window
Connect itself hangs?                   → Error 3: wrong endpoint — use the DB data VIP; cap timeout
Nothing in the UI grid either?          → auditing isn't writing to the DB (§1 prerequisites)

4. Schema & Query Recipes

Recipes are shown with the VAST DB SDK (ibis/pandas). The same filters map directly onto the VMS REST API (fields + filters, see §2) and onto the Web UI Query Panel's conditions.

The columns you'll actually use

The audit table has around 50 columns. These are the ones most queries touch:

The full column list is in ../db-audit-qry/audit-schema.md.

S3-specific columns

On protocol = 'S3' rows, these are also available (the nfs*_ / smb_* columns are null):

A note on field names (three different spellings)

The same field appears under three naming conventions depending on where you look:

When querying, always use the snake_case column names.

The nested path column

path is a struct: row(path varchar, phandle row(...)). The readable path string is the inner .path field.

# SDK (ibis predicate)
from ibis import _
predicate = _.path.path.startswith("/projects/")

# After read_all() → pandas, the column comes back as a dict:
df["full_path"] = df["path"].apply(lambda p: p["path"] if isinstance(p, dict) else None)

For the VMS API path, filter on the flat view_path column instead (e.g. /bgolliher/mputest).

rpc_type value reference

Values are case-sensitive and version-specific, so treat these as starting points and confirm against your own data (one query, shown below).

• NFSv3 / NFSv4 (UPPERCASE): CREATE, OPEN, WRITE, READ, SETATTR, MKDIR, REMOVE, RMDIR, RENAME, LINK, SYMLINK, READDIR, GETATTR, ACCESS.

• SMB (UPPERCASE): CREATE, WRITE, READ, SET_INFO, CLOSE, IOCTL, QUERY_DIRECTORY, SESSION_SETUP, LOGOFF.

• S3: spelling varies by version. On VAST 5.4/5.5, the stored values are UPPER_SNAKE: PUT_OBJECT, GET_OBJECT, COPY_OBJECT, COMPLETE_MULTIPART_UPLOAD, DELETE_OBJECT. VAST's Audited Protocol Operations doc lists them in AWS CamelCase (PutObject, ...); the audit table stores VAST's internal spelling.

• VAST Database (UPPER_SNAKE_CASE): INSERT_ROWS, UPDATE_ROWS, DELETE_ROWS, IMPORT_DATA, SELECT, QUERY_DATA, CREATE_TABLE, DROP_TABLE, ALTER_TABLE.

List the real values from your data with the distinct-values query in §3, or:

python examples/first_query.py --distinct-ops

Recipe queries (VAST DB SDK)

Each assumes an open table handle:

import vastdb
from ibis import _
with vastdb.connect(endpoint=EP, access=AK, secret=SK).transaction() as tx:
    table = tx.audit_log()   # the AuditDB, by its built-in well-known name
    ...                      # recipes below run inside this block

One habit to keep: pass limit_rows unless you want every matching row — on a busy cluster, an unbounded window holds millions of rows (§3 Error 3).

Fuller implementations (CSV/JSON export, path aggregation) are in ../db-audit-qry/. And for the most common report of all — everything one protocol did in a time window — examples/audit_timeslice.py is ready to run:

python examples/audit_timeslice.py --s3 --minutes 60 --filter getObject,putObject
python examples/audit_timeslice.py --nfs3 --filter ACCESS
python examples/audit_timeslice.py --nfsv4.1 --start '2026-07-15 10:00' --json | jq .

It normalizes operations’ spellings to the stored UPPER_SNAKE form, matches every protocol spelling variant, excludes the audit noise, and caps the fetch — the habits this section teaches, in one place.

Recipe 1: Recent activity for one user

import ibis
pred = (_.login_name == "jsmith") & (_.time >= ibis.literal(one_day_ago, type="timestamp"))
rows = table.select(
    columns=["time", "protocol", "rpc_type", "path", "status", "num_bytes"],
    predicate=pred, limit_rows=10_000,
).read_all().to_pandas()

Recipe 2: Bytes written by path prefix (depth-3 folders)

Query WRITE rows, then drop the noise paths and aggregate the truncated path in pandas (negated prefix filters don't push down, so the exclusion is client-side — §3 2c):

pred = (_.rpc_type == "WRITE") & (_.time >= ibis.literal(week_ago, type="timestamp"))
df = table.select(columns=["path", "num_bytes", "num_ops"], predicate=pred,
                  limit_rows=1_000_000).read_all().to_pandas()

df["full"] = df["path"].apply(lambda p: p.get("path") if isinstance(p, dict) else None)
df = df[~df["full"].str.startswith(("(null)", "/.vast_audit_dir/"), na=True)]
df["prefix"] = df["full"].str.split("/").str[:4].str.join("/")   # /a/b/c depth-3
agg = df.groupby("prefix").agg(bytes=("num_bytes", "sum"), ops=("num_ops", "sum")) \
        .sort_values("bytes", ascending=False)

The full version is ../db-audit-qry/audit_writes_by_path.py. For S3 writes, filter rpc_type == "PUT_OBJECT" (5.4/5.5) rather than "WRITE".

Recipe 3: Top writers across everyone (last 7 days)

pred = (_.rpc_type.startswith("WRITE")) & (_.time >= ibis.literal(week_ago, type="timestamp")) \
       & (_.login_name != "audit")
df = table.select(columns=["login_name", "uid", "num_bytes"], predicate=pred).read_all().to_pandas()
df["user"] = df["login_name"].where(df["login_name"] != "", df["uid"].astype(str))
top = df.groupby("user")["num_bytes"].sum().sort_values(ascending=False).head(50)

Recipe 4: Who deleted or renamed something under a path

pred = (_.rpc_type.isin(["REMOVE", "RMDIR", "RENAME", "DELETE_OBJECT"])) \
       & (_.path.path.startswith("/projects/")) \
       & (_.time >= ibis.literal(month_ago, type="timestamp"))
rows = table.select(
    columns=["time", "login_name", "rpc_type", "path", "rename_path", "name"],
    predicate=pred, limit_rows=10_000,
).read_all().to_pandas()

S3 delete/rename spelling is version-specific; confirm with --distinct-ops before relying on DELETE_OBJECT.

5. Advanced: SDK Access & Path Scoping

The VAST DB SDK reads the audit table the same way it reads any VAST Database table: over S3, with an access key. Use it for bulk extracts, pipelines, and embedding audit queries in tooling. This section walks through the setup, then shows how to scope what different users see by path.

Connect and query

Three steps: install, point at the data endpoint, open the audit table by name.

pip install vastdb pyarrow pandas ibis-framework
export VASTDB_ENDPOINT='http://<cluster-db-data-vip>'   # S3/tabular data VIP, not the VMS address
export VASTDB_ACCESS_KEY='<access-key>'
export VASTDB_SECRET_KEY='<secret-key>'

import os, vastdb
from ibis import _

session = vastdb.connect(endpoint=os.environ["VASTDB_ENDPOINT"],
                         access=os.environ["VASTDB_ACCESS_KEY"],
                         secret=os.environ["VASTDB_SECRET_KEY"])
with session.transaction() as tx:
    table = tx.audit_log()                # the AuditDB, by its built-in well-known name
    batch = table.select(
        columns=["time", "login_name", "protocol", "rpc_type", "path", "status"],
        predicate=(_.login_name != "audit"),
        limit_rows=100,
    ).read_all()
    print(batch.to_pandas())

That's the whole pattern. tx.audit_log() returns a regular table handle, so everything in §4 (predicates, the nested path column, recipes) applies as-is. The starter examples/first_query.py wraps this with time and path filters; examples/audit_timeslice.py builds it out into a full per-protocol report (NFSv3, NFSv4, or S3, one time window).

One habit for command-line tools: pass timeout= and backoff_config=BackoffConfig(max_tries=1) (from vastdb.config) to vastdb.connect() so a mistyped endpoint fails in seconds instead of retrying for a minute (§3 Error 3).

The AuditDB is a managed bucket

The audit table lives at fixed coordinates on every cluster — vast-audit-log-bucket / vast_audit_log_schema / vast_audit_log_table — and the SDK knows them (tx.audit_log(), just like tx.catalog() for the VAST Catalog). Because the bucket is managed by the cluster rather than backed by a view you created, it behaves differently from your own DB buckets in one visible way: it doesn't answer S3 bucket calls. It won't appear in ListBuckets (aws s3 ls), and tx.bucket("vast-audit-log-bucket") fails with MissingBucket because that call begins with an S3 HEAD Bucket check. This is normal — open it by name and query it (§3 Error 1).

On an SDK too old to have tx.audit_log() (it's present in vastdb 2.x) — or in a tool that keeps the coordinates configurable, as examples/audit_timeslice.py does — construct the handle by name explicitly. It's the same open-by-name operation:

import vastdb.bucket, vastdb.schema
b = vastdb.bucket.Bucket("vast-audit-log-bucket", tx)
table = vastdb.schema.Schema("vast_audit_log_schema", b).table("vast_audit_log_table")

Grant access: the identity policy

The querying user needs an S3 access key and one read-only identity policy with the tabular actions. That's the entire permission model for the SDK path — the audit bucket has no View Policy to join, because it isn't backed by a view.

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "ListTabular", "Effect": "Allow",
      "Action": "s3:TabularList*",
      "Resource": ["vast-audit-log-bucket", "vast-audit-log-bucket/vast_audit_log_schema/*"] },
    { "Sid": "QueryTabular", "Effect": "Allow",
      "Action": "s3:TabularQueryData",
      "Resource": ["vast-audit-log-bucket/vast_audit_log_schema/*"] }
  ]
}

• s3:TabularList* lets the SDK resolve the schema and table; s3:TabularQueryData performs the read. Both are needed for a working read-only client.

• Resource forms (from VAST's Managing Permissions for Accessing VAST Tabular Databases): "<bucket>" for bucket-level ops, "<bucket>/<schema>/*" for schema-level, "<bucket>/<schema>/<table>" for table-level. The arn:aws:s3:::<bucket>/... form is also accepted on some versions.

• This grant is table-level, so everyone who has it can query every row. To give a user only their paths, see Scoping below.

Set up order, all with vastpy (see the cheat sheet): create the user, create their access key, create the identity policy, attach it to the user.

Scoping what users see by path

The goal: UserA sees only /bgolliher/*, UserB sees only their own subtree.

The clean way to do this is a small mediated query service: a thin service that holds one privileged credential and, for each request, adds that caller's allowed path prefix to the query filter. Callers never query the table directly, so they can only ever see their own paths.

• Build it on the VMS REST API (vastauditlog/query_data with a view_path filter), which works with just VMS credentials, or on the SDK (a path.path.startswith(prefix) predicate).

• Your service is the enforcement point: map each authenticated user to their allowed prefixes, reject anything else, and add the prefix to filters server-side.

# sketch: inside your service, per authenticated request
allowed_prefix = lookup_prefix(request.user)          # e.g. "/bgolliher/"
resp = client.vastauditlog.query_data.post(
    limit=200,
    fields=["time", "login_name", "rpc_type", "view_path", "num_bytes"],
    filters={"view_path": [{"startswith": allowed_prefix}]},   # confirm operator name on your version
)

Two design notes worth knowing:

• The file view policy governs file and bucket visibility, not which audit rows a query returns. Row filtering belongs in your service.

• The s3:TabularQueryData grant is table-level, so the service, not the policy, is where per-user scoping happens.

This model is service-enforced (the boundary is your code), which fits most "scope results to the relevant team" needs.

Out of scope: Row and Column Security

VAST also offers VAST Database Row and Column Security: identity-policy statements ("Effect":"RowColumnSecurity") that make the database itself filter rows and columns per user. It's the strongest boundary, but it's enforced only by Trino query engines and needs end-user impersonation, so it's out of scope for this guide. If you later need a hard database-enforced boundary, see VAST's VAST Database Row and Column Security doc.

6. Appendix: CLI, Config Links & Verification

Environment variables (SDK / companion scripts)

The examples/ scripts and the ../db-audit-qry/ tools read these:

export VASTDB_ENDPOINT='http://<cluster-vip>'   # VAST DB data endpoint (not VMS)
export VASTDB_ACCESS_KEY='<access-key>'
export VASTDB_SECRET_KEY='<secret-key>'

The audit bucket/schema/table names are fixed (vast-audit-log-bucket / vast_audit_log_schema / vast_audit_log_table) and built into the SDK, so they never need to be configured.

vastpy cheat-sheet

vastpy is a thin VMS REST client (SDK and vastpy-cli). It's a schema-less passthrough: vastpy-cli <get|post|patch|delete> <endpoint> key=value .... Values that look like JSON are parsed as JSON.

# --- Auth (env or per-call flags) ---
export VMS_ADDRESS=vms.example.com
export VMS_USER=admin
export VMS_PASSWORD='********'
# or:  export VMS_TOKEN=<api-token>     # VAST 5.3+

# --- Users & keys ---
vastpy-cli post users name='auditor' full_name='Audit Analyst' user_type='LOCAL'
vastpy-cli post users/<user_id>/access_keys          # or .../s3_access_keys on some versions

# --- Identity (S3) policies ---
vastpy-cli get  s3policies
vastpy-cli post s3policies name='auditdb-readonly' policy="$(cat policy.json)" tenant_id=1
vastpy-cli patch s3policies/<policy_id> users='["auditor"]'   # attach user (version-dependent)

# --- Audit config (admin; full enable procedure in §0) ---
vastpy-cli get   clusters                              # find cluster id
vastpy-cli get   clusters/<cluster_id>/auditing        # current settings
vastpy-cli patch clusters/<cluster_id>/auditing --file-input audit-config.json

# --- Query audit data via the VMS API (alternative to the SDK) ---
# Use POST, and always include `filters` (the endpoint requires it).
vastpy-cli post vastauditlog/query_data limit=20 \
  filters='{"s3_bucket_name":[{"eq":"mputest"}]}' \
  fields='["time","login_name","rpc_type","view_path","num_bytes"]'

Example audit-config.json (enables auditing and routes to VAST DB; see §0 for the full field set and protocol selection):

{
  "read_access_users": ["vastadmin"],
  "read_access_users_groups": ["vastadmin"],
  "protocols_audit": {
    "log_full_path": true,
    "modify_data_md": true,
    "create_delete_files_dirs_objects": true
  },
  "enable_vast_db_audit": true
}

A note on TLS: vastpy doesn't verify certificates unless you pass --cert-file / cert_file, and the same goes for vastdb.connect(..., ssl_verify=False). Fine for labs; set a CA bundle in production.

VAST DB tabular permissions

Tabular operations are granted by an identity policy with s3:Tabular* actions, scoped by Resource: "<bucket>" (bucket-level, e.g., list schemas), "<bucket>/*" (object-level), "<bucket>/<schema>/*" (schema-level), "<bucket>/<schema>/<table>" (table-level).

For the AuditDB (a managed bucket, like the VAST Catalog), the identity policy is the whole model — there is no view or View Policy involved (§5). For DB buckets you create yourself (S3-enabled DATABASE views), the user must additionally be in the view's View Policy with bucket-listing permission for the bucket to be visible (VAST Creating View Policies, step 6).

Common tabular actions (identity-policy Action values, all s3:-prefixed):

A read-only audit client needs s3:TabularList* plus s3:TabularQueryData. Setup order: create the user and access/secret key, create the identity policy, assign it to the user.

Smoke-testing SDK access

The audit table's coordinates are fixed and built into the SDK, so the smoke test is one call:

import vastdb
with vastdb.connect(endpoint=EP, access=AK, secret=SK).transaction() as tx:
    t = tx.audit_log()
    print(t.select(columns=["time", "protocol", "rpc_type"], limit_rows=5)
           .read_all().to_pandas())

Remember that the audit bucket won't show up in aws s3 ls or answer tx.bucket() — That's expected for a managed bucket (§5).

Official VAST configuration docs

The configuration side (turning auditing on, choosing what to audit) is covered by these VAST KB pages; link to your customer's versioned copies:

Verification checklist

Items below marked as verified were confirmed live on VAST 5.5.0.1 and/or 5.4.4.0 clusters. Anything version-sensitive is worth re-confirming on your own cluster before you rely on it.

Verified live (5.5.0.1 and 5.4.4.0):

• The VMS API path works with just VMS credentials: vastauditlog/query_data via vastpy returned real rows on both versions. Params used: limit, fields (snake_case columns), and filters ({"<field>":[{"eq":<v>}]}; the filters field is required).

• The SDK path works with just an S3 key and identity policy (verified on 5.4.4.0, vastdb 2.0.14): tx.audit_log() opened the table and predicate queries returned rows. No DATABASE view for the audit bucket exists or is needed; tx.bucket() / ListBuckets correctly do not see it.

• Predicate pushdown is fast (a time+protocol+rpc_type filter with limit_rows=20 returned in under a second), but an unbounded read_all() over a busy window can run for many minutes — always cap or batch (§3 Error 3).

• Pushdown supports equality, ranges, isin, isnull, and positive startswith (including on the nested path.path), but not negated startswith (~_.x.startswith(...) raises NotImplementedError on vastdb 2.0.14) — apply noise-exclusion filters client-side (§3 2c).

• rpc_type values for S3 are stored as UPPER_SNAKE on both versions: PUT_OBJECT, GET_OBJECT, COPY_OBJECT, COMPLETE_MULTIPART_UPLOAD. List the distinct values on your own cluster before writing filters. (Before 5.4.1, DeleteByLifecycleRule is recorded as DeleteObject.)

• status holds Success or the protocol's error name (observed: NFS3ERR_STALE, NoSuchKey); cnode_name records which CNode served each request.

• The S3/tabular data VIP and the VMS management host are different addresses. vastpy talks to the management host; pointing it at the S3 endpoint returns 400 CredentialsNotSupported.

• vastdb.connect() has no request timeout and retries for about a minute by default, so a wrong endpoint looks like a hang. The example scripts pass timeout= and backoff_config=BackoffConfig(max_tries=1) to fail fast (§3 Error 3).

Confirm on your cluster:

• VMS API filter operators beyond eq (such as startswith for view_path), from https://<vms>/docs/.

• The vastpy key-creation sub-resource (access_keys vs s3_access_keys) and the policy-attachment call on your version.

Row and Column Security (a Trino-enforced per-user boundary) is out of scope for this guide; see VAST's VAST Database Row and Column Security doc if you ever need a hard database-enforced boundary.

#!/usr/bin/env python3
"""
audit_timeslice.py: show AuditDB records for one protocol in a time window.

Queries the audit table via the vastdb SDK, opening the managed audit bucket by its
well-known name (like tx.audit_log(); see ../05-advanced-sdk-access.md). The querying
user just needs an identity policy with s3:TabularList* + s3:TabularQueryData.

SETUP
    pip install vastdb pyarrow pandas ibis-framework
    # Either export these...
    export VASTDB_ENDPOINT='http://<cluster-vip>'   # VAST DB data endpoint, NOT the VMS address
    export VASTDB_ACCESS_KEY='<access-key>'
    export VASTDB_SECRET_KEY='<secret-key>'
    # ...or pass --endpoint/--access-key/--secret-key. Missing config fails immediately;
    # an unreachable endpoint gives up after --timeout seconds (default 5).

USAGE
    python audit_timeslice.py --s3                                  # last 15 min of S3 activity
    python audit_timeslice.py --s3 --filter getObject,putObject
    python audit_timeslice.py --nfs3 --minutes 60 --filter ACCESS
    python audit_timeslice.py --nfsv4.1 --start '2026-07-15 10:00' --filter open
    python audit_timeslice.py --s3 --json | jq .
"""
import argparse
import os
import re
import sys
from datetime import datetime, timedelta, timezone

try:
    import ibis
    import vastdb
    from ibis import _
    from vastdb.config import BackoffConfig
except ImportError:
    sys.exit("Missing deps. Run: pip install vastdb pyarrow pandas ibis-framework")

# The stored `protocol` spelling varies by release, so match every known variant.
PROTOCOLS = {
    "nfs3": ["NFSv3", "NFS3", "NFSV3"],
    "nfsv41": ["NFSv4", "NFS4", "NFSV4", "NFSv4.1"],  # the table doesn't split 4.0/4.1
    "s3": ["S3"],
}
COLUMNS = ["time", "login_name", "uid", "client_ip",
           "protocol", "rpc_type", "path", "status", "num_bytes"]


def parse_args():
    p = argparse.ArgumentParser(
        description="Show all AuditDB records for one protocol in a time window.",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
    proto = p.add_mutually_exclusive_group(required=True)
    proto.add_argument("--nfs3", action="store_true", help="Show NFSv3 records.")
    proto.add_argument("--nfsv4.1", dest="nfsv41", action="store_true", help="Show NFSv4 records.")
    proto.add_argument("--s3", action="store_true", help="Show S3 records.")

    p.add_argument("--minutes", type=int, default=15, help="Length of the time window.")
    p.add_argument("--start", metavar="'YYYY-MM-DD HH:MM'",
                   help="Window start (UTC unless an offset is given); default is MINUTES ago.")
    p.add_argument("--filter", dest="ops", metavar="OP[,OP...]",
                   help="Only these rpc_type ops, e.g. getObject,putObject or ACCESS."
                        "Spelling is normalized to the stored UPPER_SNAKE form.")
    p.add_argument("--json", action="store_true", help="Emit JSON lines instead of a table.")
    p.add_argument("--limit", type=int, default=1000,
                   help="Max rows to fetch; busy windows can hold millions. 0 = no cap.")

    p.add_argument("--endpoint", default=os.environ.get("VASTDB_ENDPOINT"),
                   help="VAST DB data endpoint (env: VASTDB_ENDPOINT)")
    p.add_argument("--access-key", default=os.environ.get("VASTDB_ACCESS_KEY"),
                   help="S3 access key (env: VASTDB_ACCESS_KEY)")
    p.add_argument("--secret-key", default=os.environ.get("VASTDB_SECRET_KEY"),
                   help="S3 secret key (env: VASTDB_SECRET_KEY)")
    p.add_argument("--bucket", default=os.environ.get("VASTDB_AUDIT_BUCKET", "vast-audit-log-bucket"))
    p.add_argument("--schema", default=os.environ.get("VASTDB_AUDIT_SCHEMA", "vast_audit_log_schema"))
    p.add_argument("--table", default=os.environ.get("VASTDB_AUDIT_TABLE", "vast_audit_log_table"))
    p.add_argument("--insecure", action="store_true",
                   help="Disable TLS verification (self-signed lab certs only).")
    p.add_argument("--timeout", type=float, default=5,
                   help="Seconds before giving up on an unresponsive endpoint.")

    args = p.parse_args()
    missing = [n for n in ("endpoint", "access_key", "secret_key") if not getattr(args, n)]
    if missing:
        p.error("missing required config: " + ", ".join(missing)
                + " (set VASTDB_ENDPOINT / VASTDB_ACCESS_KEY / VASTDB_SECRET_KEY or pass flags)")
    return args


def normalize_op(op):
    """Map CLI-friendly spellings to the stored form: getObject -> GET_OBJECT, access -> ACCESS."""
    return re.sub(r"(?<=[a-z0-9])(?=[A-Z])", "_", op.strip()).upper()


def time_window(args):
    """Return the half-open [start, end) window, anchored at --start or trailing from now."""
    if args.start:
        start = datetime.fromisoformat(args.start)
        start = start if start.tzinfo else start.replace(tzinfo=timezone.utc)
    else:
        start = datetime.now(timezone.utc) - timedelta(minutes=args.minutes)
    return start, start + timedelta(minutes=args.minutes)


def build_predicate(args, start, end):
    flag = next(name for name in PROTOCOLS if getattr(args, name))
    predicate = (
        (_.time >= ibis.literal(start, type="timestamp"))
        & (_.time < ibis.literal(end, type="timestamp"))
        & (_.protocol.isin(PROTOCOLS[flag]))
        & (_.login_name != "audit")  # the cluster's own audit-table writes
    )
    if args.ops:
        ops = [normalize_op(op) for op in args.ops.split(",") if op.strip()]
        predicate &= _.rpc_type.isin(ops)
    return predicate


def fetch(args, predicate):
    # The SDK's defaults (no request timeout, ~60s of retries) hang for minutes on a
    # bad endpoint, so cap the timeout and disable retries.
    kwargs = dict(endpoint=args.endpoint, access=args.access_key, secret=args.secret_key,
                  timeout=args.timeout, backoff_config=BackoffConfig(max_tries=1))
    if args.insecure:
        kwargs["ssl_verify"] = False
    print(f"Connecting to {args.endpoint} (timeout {args.timeout:g}s)...", file=sys.stderr)
    try:
        session = vastdb.connect(**kwargs)
    except Exception as e:
        sys.exit(f"Could not connect to {args.endpoint}: {e}\n"
                 "Check --endpoint / $VASTDB_ENDPOINT: it must be the VAST DB data VIP "
                 "(http://main.<cluster>...), not the VMS/GUI host.")
    with session.transaction() as tx:
        try:
            # Open by well-known name, like tx.audit_log(): the managed audit bucket is not
            # an S3 view, so tx.bucket()'s HEAD-bucket pre-check 404s (MissingBucket) even
            # when tabular queries against it are allowed.
            b = vastdb.bucket.Bucket(args.bucket, tx)
            table = vastdb.schema.Schema(args.schema, b).table(args.table)
        except Exception as e:
            sys.exit(f"Could not open {args.bucket}/{args.schema}/{args.table}: {e}\n"
                     "Does your identity policy grant s3:Tabular* on this resource? "
                     "See ../03-troubleshooting.md (Error 1).")
        return table.select(columns=COLUMNS, predicate=predicate,
                            limit_rows=args.limit or None).read_all().to_pandas()


def main():
    args = parse_args()
    start, end = time_window(args)
    df = fetch(args, build_predicate(args, start, end))

    # `path` is a nested struct (a dict after to_pandas); the string is at .path.
    df["path"] = df["path"].map(lambda p: p.get("path", "") if isinstance(p, dict) else p or "")
    df = df[~df["path"].str.startswith("/.vast_audit_dir/")].sort_values("time")

    if df.empty:
        sys.exit(f"No rows between {start:%Y-%m-%d %H:%M:%S} and {end:%H:%M:%S} UTC. "
                 "Widen the window, or check the ops with first_query.py --distinct-ops.")
    if args.json:
        print(df.to_json(orient="records", lines=True, date_format="iso"), end="")
    else:
        capped = " (hit --limit; not the full window)" if args.limit and len(df) >= args.limit else ""
        print(f"{len(df)} row(s){capped}, {start:%Y-%m-%d %H:%M:%S} -> {end:%H:%M:%S} UTC:\n")
        print(df.to_string(index=False))


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""
first_query.py: a clean, minimal starting point for querying the VAST AuditDB.

WHAT IT DOES
    1. Connects to the VAST DB data endpoint with an S3 access/secret key.
    2. Opens the audit table by its built-in well-known name: tx.audit_log().
       (The AuditDB is a managed bucket; it doesn't appear in bucket listings.)
    3. Shows the two things people most often get wrong:
         - the nested `path` column (the string is at path.path)
         - protocol-specific, case-sensitive rpc_type values
    4. Demonstrates a path-scoped query.

SCOPING NOTE
    The `--path` filter below is applied by this script (client-side); the SDK grant
    itself is table-level. To enforce per-user path visibility (UserA sees
    /bgolliher/*, UserB can't), front the AuditDB with a mediated service that injects
    each caller's allowed prefix; see ../05-advanced-sdk-access.md (Scoping what users
    see by path).

SETUP
    pip install vastdb pyarrow pandas ibis-framework

    export VASTDB_ENDPOINT='http://<cluster-vip>'   # VAST DB data endpoint, NOT the VMS address
    export VASTDB_ACCESS_KEY='<access-key>'
    export VASTDB_SECRET_KEY='<secret-key>'

USAGE
    python first_query.py                                  # last 24h, all paths, 20 rows
    python first_query.py --hours 6 --limit 50
    python first_query.py --path /bgolliher/ --hours 168   # last 7 days under a path
    python first_query.py --distinct-ops                   # list the rpc_type values you actually have
"""
import argparse
import os
import sys
from datetime import datetime, timedelta, timezone

try:
    import vastdb
    import ibis
    from ibis import _
except ImportError:
    sys.exit("Missing deps. Run: pip install vastdb pyarrow pandas ibis-framework")


def parse_args():
    p = argparse.ArgumentParser(
        description="Minimal VAST AuditDB query starter.",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
    p.add_argument("--endpoint", default=os.environ.get("VASTDB_ENDPOINT"),
                   help="VAST DB data endpoint, e.g. http://<cluster-vip> (env: VASTDB_ENDPOINT)")
    p.add_argument("--access-key", default=os.environ.get("VASTDB_ACCESS_KEY"),
                   help="S3 access key (env: VASTDB_ACCESS_KEY)")
    p.add_argument("--secret-key", default=os.environ.get("VASTDB_SECRET_KEY"),
                   help="S3 secret key (env: VASTDB_SECRET_KEY)")
    p.add_argument("--hours", type=int, default=24, help="Look back this many hours.")
    p.add_argument("--path", default="", help="Only rows whose path starts with this prefix (client-side).")
    p.add_argument("--limit", type=int, default=20, help="Max rows to print.")
    p.add_argument("--distinct-ops", action="store_true",
                   help="Instead of rows, print the distinct (protocol, rpc_type) values present.")
    p.add_argument("--insecure", action="store_true",
                   help="Disable TLS verification (self-signed lab certs only).")
    args = p.parse_args()
    missing = [n for n in ("endpoint", "access_key", "secret_key") if not getattr(args, n)]
    if missing:
        p.error("missing required config: " + ", ".join(missing) +
                " (set VASTDB_ENDPOINT / VASTDB_ACCESS_KEY / VASTDB_SECRET_KEY or pass flags)")
    return args


def connect(args):
    kwargs = dict(endpoint=args.endpoint, access=args.access_key, secret=args.secret_key)
    if args.insecure:
        kwargs["ssl_verify"] = False
    try:
        return vastdb.connect(**kwargs)
    except Exception as e:
        sys.exit(f"Failed to connect to {args.endpoint}: {e}")


def get_table(tx):
    """Open the AuditDB by its built-in well-known name, with a helpful message on failure."""
    try:
        # NOT tx.bucket("vast-audit-log-bucket"): the AuditDB is a managed bucket that
        # doesn't answer S3 bucket calls, so tx.bucket() raises MissingBucket by design.
        return tx.audit_log()
    except Exception as e:
        sys.exit(
            f"\nCould not open the audit table: {e}\n"
            "Check, in order:\n"
            "  - Does your user have an identity policy allowing s3:TabularList* and\n"
            "    s3:TabularQueryData on vast-audit-log-bucket, and is it attached?\n"
            "  - Is 'Save audit logs to VAST DB' enabled? (vast_audit_log_state: CREATED)\n"
            "  - Is VASTDB_ENDPOINT the VAST DB data VIP (not the VMS address)?\n"
            "Tip: the Web UI Query Panel and VMS API path (examples/query_vms_api.py) work\n"
            "with just VMS credentials. See ../03-troubleshooting.md (Error 1)."
        )


def path_str(cell):
    """The `path` column is a nested struct; the string is at .path (a dict after to_pandas)."""
    if isinstance(cell, dict):
        return cell.get("path") or ""
    return cell or ""


def main():
    args = parse_args()
    session = connect(args)

    since = datetime.now(timezone.utc) - timedelta(hours=args.hours)

    with session.transaction() as tx:
        table = get_table(tx)

        if args.distinct_ops:
            # The single most useful troubleshooting query: what rpc_type values do I actually have?
            # Sampled (not exhaustive): an uncapped scan of a busy audit table takes minutes.
            df = table.select(columns=["protocol", "rpc_type"],
                              limit_rows=200_000).read_all().to_pandas()
            counts = df.value_counts(["protocol", "rpc_type"]).reset_index(name="count")
            print(f"\nDistinct (protocol, rpc_type) values in a {len(df)}-row sample:")
            print(counts.to_string(index=False))
            print("\nrpc_type spelling is version-specific: NFS/SMB are UPPERCASE, VAST DB is "
                  "UPPER_SNAKE, and S3 is UPPER_SNAKE on 5.5 (PUT_OBJECT) but CamelCase "
                  "(PutObject) in VAST's op-list doc. Trust the values printed above.")
            return

        # Time window + drop the audit service account. (The other noise filters are
        # negated prefix matches, which don't push down — they're applied client-side below.)
        predicate = (
            (_.time >= ibis.literal(since, type="timestamp"))
            & (_.login_name != "audit")
        )
        if args.path:
            # Server-side prefix filter on the nested path string (positive prefix: pushes down).
            predicate = predicate & (_.path.path.startswith(args.path))

        cols = ["time", "login_name", "uid", "client_ip",
                "protocol", "rpc_type", "path", "status", "num_bytes"]
        # Cap the fetch: a busy window can hold millions of rows (../03, Error 3).
        fetch_cap = max(args.limit, 100_000)
        df = table.select(columns=cols, predicate=predicate,
                          limit_rows=fetch_cap).read_all().to_pandas()
        capped = len(df) >= fetch_cap

    # Client-side noise exclusion (negated prefix filters can't be pushed down).
    df["path"] = df["path"].map(path_str)
    df = df[~df["path"].str.startswith(("(null)", "/.vast_audit_dir/"))]

    if df.empty:
        print(f"\nNo rows in the last {args.hours}h"
              + (f" under '{args.path}'" if args.path else "") + ".")
        print("If you expected data: check rpc_type casing, the time window, and the path prefix "
              "(see ../03-troubleshooting.md, Error 2).")
        return

    df = df.sort_values("time", ascending=False).head(args.limit)
    note = " (sampled: fetch cap hit, so these may not be the newest)" if capped else ""
    print(f"\nMost recent {len(df)} of matching rows{note}"
          + (f" under '{args.path}'" if args.path else "") + ":\n")
    with_cols = ["time", "login_name", "protocol", "rpc_type", "path", "status", "num_bytes"]
    print(df[with_cols].to_string(index=False))


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""
query_vms_api.py: query the VAST AuditDB through the VMS REST API (vastpy).

Reads the audit table through the management plane, so it works with just VMS
credentials, as soon as auditing is on. Verified on VAST 5.5.0.1. (For the SDK path
and per-user path scoping, see ../05-advanced-sdk-access.md.)

SETUP
    pip install vastpy
    export VMS_ADDRESS='var204.selab.vastdata.com'   # VMS/GUI host (bare), not the S3 endpoint
    export VMS_USER='admin'
    export VMS_PASSWORD='********'
    # or:  export VMS_TOKEN='<api-token>'   (VAST 5.3+)

USAGE
    python query_vms_api.py --bucket mputest --top 10
    python query_vms_api.py --path /bgolliher/ --top 20
    python query_vms_api.py --user blake.golliher --op PUT_OBJECT --top 50

NOTES
    The query_data endpoint requires a 'filters' field, so pass at least one of
    --bucket/--user/--op/--path. The 'eq' operator is confirmed; 'startswith' (used
    for --path) may vary by version; see https://<vms>/docs/. rpc_type values are
    version-specific (on 5.5.0.1, S3 writes are PUT_OBJECT, not PutObject); run with
    a broad filter first to see what you have.
"""
import argparse
import os
import sys

try:
    import urllib3
    urllib3.disable_warnings()
    from vastpy import VASTClient
except ImportError:
    sys.exit("Missing deps. Run: pip install vastpy")


DEFAULT_FIELDS = ["time", "login_name", "uid", "client_ip", "protocol",
                  "rpc_type", "view_path", "num_bytes", "status"]


def build_client():
    address = os.environ.get("VMS_ADDRESS")
    if not address:
        sys.exit("$VMS_ADDRESS is required (the VMS/GUI hostname, not the S3 data endpoint).")
    if address.startswith(("http://", "https://")):
        sys.exit("$VMS_ADDRESS must be a bare hostname; vastpy adds https:// itself.")
    kwargs = {"address": address}
    if os.environ.get("VMS_TOKEN"):
        kwargs["token"] = os.environ["VMS_TOKEN"]
    else:
        user, pwd = os.environ.get("VMS_USER"), os.environ.get("VMS_PASSWORD")
        if not (user and pwd):
            sys.exit("Set $VMS_USER and $VMS_PASSWORD (or $VMS_TOKEN).")
        kwargs["user"], kwargs["password"] = user, pwd
    if os.environ.get("VMS_TENANT_NAME"):
        kwargs["tenant_name"] = os.environ["VMS_TENANT_NAME"]
    return VASTClient(**kwargs)


def build_filters(args):
    filters = {}
    if args.bucket:
        filters["s3_bucket_name"] = [{"eq": args.bucket}]
    if args.user:
        filters["login_name"] = [{"eq": args.user}]
    if args.op:
        filters["rpc_type"] = [{"eq": args.op}]
    if args.path:
        # 'startswith' operator may differ by version; see module docstring.
        filters["view_path"] = [{"startswith": args.path}]
    return filters


def normalize(resp):
    """query_data returns {'prop_list': [...], 'results': [[...], ...]}. Yield dict rows."""
    if not isinstance(resp, dict) or "results" not in resp:
        sys.exit(f"unexpected response: {resp!r}")
    cols = resp.get("prop_list") or DEFAULT_FIELDS
    for row in resp["results"]:
        if isinstance(row, (list, tuple)):
            yield dict(zip(cols, row))
        elif isinstance(row, dict):
            yield row
        else:
            yield {"value": row}


def main():
    p = argparse.ArgumentParser(description="Query the VAST AuditDB via the VMS REST API.")
    p.add_argument("--bucket", help="Filter by S3 bucket name (exact).")
    p.add_argument("--user", help="Filter by login_name (exact).")
    p.add_argument("--op", help="Filter by rpc_type (exact), e.g. PUT_OBJECT, WRITE.")
    p.add_argument("--path", help="Filter by view_path prefix.")
    p.add_argument("--top", type=int, default=20, help="Max rows to fetch (default 20).")
    p.add_argument("--fields", help="Comma-separated columns to return (default: a useful set).")
    args = p.parse_args()

    client = build_client()
    fields = [f.strip() for f in args.fields.split(",")] if args.fields else DEFAULT_FIELDS

    try:
        resp = client.vastauditlog.query_data.post(
            limit=args.top, fields=fields, filters=build_filters(args),
        )
    except Exception as e:
        hint = ("Check that $VMS_ADDRESS is the management host (not the S3 endpoint) "
                "and that auditing is enabled. See ../03-troubleshooting.md.")
        if "filters" in str(e).lower():
            hint = ("The query_data endpoint requires a non-empty 'filters'. Pass a filter, "
                    "e.g. --bucket mputest (or --user/--op/--path).")
        sys.exit(f"query_data failed: {e}\n{hint}")

    rows = list(normalize(resp))
    if not rows:
        print("No matching audit rows. Loosen filters, or run without --op to see real rpc_type values.")
        return

    print(f"{len(rows)} row(s):\n")
    for r in rows:
        print("  " + "  ".join(f"{k}={r.get(k)}" for k in fields if k in r))


if __name__ == "__main__":
    main()