The VAST Database supports a number of query engines, including Spark and Trino. It also supports the VAST Query Engine. This query engine provides improved performance for VAST Database table operations using an ADBC Client Library.
The VAST Query Engine is fully embedded within VAST CNodes.
VAST Query Engine Features
The VAST Query Engine supports the following:
Vector search. The ability to query vectors in the database for the nearest neighbors.
Column Allow/Deny. The ability to restrict access to columns and rows in a database based on a users identity.
Filter pushdown. This is an optimized scan/query of selected rows and/or columns in a database table.
Nested data types. The ability to nest datatypes such as
struct,array.Aggregations. The ability to perform aggregation functions on query results. Examples of aggregation functions:
SUM, AVG, MIN, MAX.
Supported SQL Commands
The VAST Query Engine supports these SQL DDL commands.
Command | Description |
|---|---|
CREATE SCHEMA [IF NOT EXISTS] <schema-full-name> | Creates a schema, the full name includes the bucket and all nested schema directories Options:
|
DROP SCHEMA <schema-full-name> | Deletes an existing schema |
CREATE [OR REPLACE] TABLE [IF NOT EXISTS] <table-full-name> (<col-name> <col-type> …) | Creates a regular table; full name includes bucket, nested schema directories, and table name Options:
|
CREATE [OR REPLACE] TABLE [IF NOT EXISTS] <table-full-name> (… CONSTRAINT <constraint-name> SORT KEY (col1, col2, …)) | Creates a table with an Elysium sort key constraint (up to 4 sort columns) Options:
|
DROP TABLE <table-full-name> | Drops an existing table |
DESCRIBE [TABLE] <table-full-name> | Displays the column definitions of a table |
ALTER TABLE <table-full-name> RENAME TO <new-table-full-name> | Renames a table |
ALTER TABLE <table-full-name> RENAME COLUMN <col-name> TO <new-col-name> | Renames a column in a table |
ALTER TABLE <table-full-name> ADD [COLUMN] <col-name> <col-type> | Adds a new column to a table |
ALTER TABLE <table-full-name> DROP [COLUMN] <col-name> | Removes a column from a table |
ALTER TABLE <table-full-name> ADD CONSTRAINT <constraint-name> SORT KEY (col1, col2, …) | Enables column sorting by adding a sort key constraint to an existing table. See Sorted Tables |
These SQL DML commands are supported:
Command | Description |
|---|---|
INSERT INTO <table-full-name> VALUES (col1-value, col2-value, …) | Inserts a row; all columns must be provided in table order |
INSERT INTO <table-full-name> (col1-name, col2-name, …) VALUES (col1-value, col2-value, …) | Inserts a row with values for a specified subset of columns |
INSERT INTO <table-full-name> SELECT … | Inserts rows produced by a SELECT query into the target table |
UPDATE <table-full-name> SET col1-name = col1-value, col2-name = col2-value WHERE condition | Updates one or more column values in rows that match the specified condition |
DELETE FROM <table-full-name> WHERE condition | Deletes rows from a table that match the specified condition |
INSERT INTO <table-full-name> (part_colname_1, partition_colname_2, partition_colname_3, partition_colname_4, $parquet_file_path") VALUES (value1_1, value2_1, value3_1, parquet_file_path_1), (value1_2, value2_2, value3_2, parquet_file_path_2),... | Inserts data from parquet files into a partitioned table, according to the partition columns |
Supported Aggregation Functions
The VAST Query Engine supports these aggregation functions:
Function | Example | Expected result (short explanation) |
|---|---|---|
any_value(x)
| SELECT any_value(country) FROM t; | Returns the first non‑NULL country encountered (order‑sensitive). |
avg(x) | SELECT avg(salary) FROM t; | Average of non‑NULL salaries; NULLs are ignored; empty group → NULL. |
mean(x) | SELECT mean(salary) FROM t; | Average of non‑NULL salaries; NULLs are ignored; empty group → NULL. |
sum(x) | SELECT sum(amount) FROM t; | Sum of non‑NULL amounts. For BOOLEAN, counts TRUEs. |
min(x)
| SELECT min(score) FROM t; | Minimum non‑NULL score. |
max(x)
| SELECT max(score) FROM t; | Maximum non‑NULL score. |
bool_and(x) | SELECT bool_and(ok) FROM t; | TRUE if all non‑NULL booleans are TRUE; FALSE otherwise. |
bool_or(x) | SELECT bool_or(ok) FROM t; | TRUE if any non‑NULL boolean is TRUE; FALSE if none are TRUE. |
count() | SELECT count() FROM t; | Row count (includes NULLs). Same as count(*). |
count(x) | SELECT count(user_id) FROM t; | Count of non‑NULL user_id values. |
countif(cond) | SELECT countif(score >= 90) FROM t; | Number of rows where condition is TRUE. |
product(x) | SELECT product(multiplier) FROM t; | Product of all non‑NULL multipliers (empty group → NULL). |
stddev_pop(x) | SELECT stddev_pop(v) FROM t; | Population standard deviation. |
stddev_samp(x) / stddev(x) | SELECT stddev_samp(v) FROM t; | Sample standard deviation (Bessel correction). |
var_pop(x) | SELECT var_pop(v) FROM t; | Population variance. |
var_samp(x) / variance(x) | SELECT var_samp(v) FROM t; | Sample variance (Bessel correction). |
approx_count_distinct(x) | SELECT approx_count_distinct(user_id) FROM t; | Approximate cardinality using HyperLogLog. |
Accessing VAST DataBases using the Query Engine
In order to access the VAST DataBase using the Query Engine, you will need the following:
A virtual IP pool dedicated to the Query Engine (see Creating Virtual IP Pools for the VAST Query Engine)
An S3 user with permission to access the DataBase, and with an access and secret key pair (the keys are used by the client application). See Managing Local Users.
An identity or bucket policy for the user, that sets the permissions to access the database. This policy must include
s3:QueryEngine*permissions for general Query Engine actions, ands3:Tabular*permissions for DDL and DML actions. See Creating Identity Policies.A client application that uses the VAST Query Engine ADBC library to access the Query Engine (see VAST DataBase Query Engine ADBC Driver for Client Applications).
Creating Virtual IP Pools for the VAST Query Engine
The Query Engine requires a dedicated virtual IP pool for client access. Create this pool with the role Query Engine, using the VAST Web UI.
Navigate to the Network Access page, and select Virtual IP Pools.
Click Create Virtual IP Pool.
Compete details for the virtual IP pool following the procedure described in Creating Virtual IP Pools, selecting role type Query Engine. Define also a FQDN in the DNS for the virtual IP pool (see Step 6).
VAST DataBase Query Engine ADBC Driver for Client Applications
VAST provides an open-source ADBC driver that you can use to provide access for your client application to the VAST Query Engine.
Download the driver from https://github.com/vast-data/vastdb-adbc-driver.
The ADBC can be used in a Linux x86_64 environment, for programming languages that have a driver manager (for example, C++, Rust, Python, Ruby, Go, and Java).
When using this driver, you will need to provide a virtual IP address and an access/secret key pair, to access databases (see Creating Virtual IP Pools for the VAST Query Engine.
ADBC User Impersonation
You can run VAST ADBC driver commands with the credentials of the end user (instead of using the service account of the VAST ADBC driver). To do so:
Grant the
s3:EndUserImpersonationpermission to the VAST ADBC driver's service account through the identity or bucket policy. For example:{ "Version": "2012-10-17", "Statement": [ { "Sid": "Allow end user impersonation on Tabular operations", "Effect": "Allow", "Action": "s3:EndUserImpersonation", "Resource": ["arn:aws:s3:::*"] } ] }Use the
vast.db.end_userconnection option to pass the username of the end user along with the query. For example:conn = adbc_driver_manager.dbapi.connect( driver="driver_path", db_kwargs={ "vast.db.endpoint": "server_ip", "vast.db.access_key": "access_key", "vast.db.secret_key": "secret_key", }, conn_kwargs={ "vast.db.end_user": "user_name", } )