Documentation Index

Fetch the complete documentation index at: https://kb.vastdata.com/llms.txt

Use this file to discover all available pages before exploring further.

VAST Query Engine

Prev Next

The VAST Database supports a number of query engines, including Spark and Trino. It also supports the VAST Query Engine. This query engine provides improved performance for VAST Database table operations using an ADBC Client Library.

The VAST Query Engine is fully embedded within VAST CNodes.

VAST Query Engine Features

The VAST Query Engine supports the following:

  • Vector search. The ability to query vectors in the database for the nearest neighbors.

  • Column Allow/Deny. The ability to restrict access to columns and rows in a database based on a users identity.

  • Filter pushdown. This is an optimized scan/query of selected rows and/or columns in a database table.

  • Nested data types. The ability to nest datatypes such as struct, array.

  • Aggregations. The ability to perform aggregation functions on query results. Examples of aggregation functions: SUM, AVG, MIN, MAX.

Supported SQL Commands

The VAST Query Engine supports these SQL DDL commands.

Command

Description

CREATE SCHEMA [IF NOT EXISTS] <schema-full-name>

Creates a schema, the full name includes the bucket and all nested schema directories

Options:

  • IF NOT EXISTS. Create the schema only if it does not yet exist

DROP SCHEMA <schema-full-name>

Deletes an existing schema

CREATE [OR REPLACE] TABLE [IF NOT EXISTS] <table-full-name> (<col-name> <col-type> …)

Creates a regular table; full name includes bucket, nested schema directories, and table name

Options:

  • OR REPLACE. Replace the table with a new one if it exists

  • IF NOT EXISTS. Create the table only if it does not yet exist

CREATE [OR REPLACE] TABLE [IF NOT EXISTS] <table-full-name> (… CONSTRAINT <constraint-name> SORT KEY (col1, col2, …))

Creates a table with an Elysium sort key constraint (up to 4 sort columns)

Options:

  • OR REPLACE. Replace the table with a new one if it exists

  • IF NOT EXISTS. Create the table only if it does not yet exist

DROP TABLE <table-full-name>

Drops an existing table

DESCRIBE [TABLE] <table-full-name>

Displays the column definitions of a table

ALTER TABLE <table-full-name> RENAME TO <new-table-full-name>

Renames a table

ALTER TABLE <table-full-name> RENAME COLUMN <col-name> TO <new-col-name>

Renames a column in a table

ALTER TABLE <table-full-name> ADD [COLUMN] <col-name> <col-type>

Adds a new column to a table

ALTER TABLE <table-full-name> DROP [COLUMN] <col-name>

Removes a column from a table

ALTER TABLE <table-full-name> ADD CONSTRAINT <constraint-name> SORT KEY (col1, col2, …)

Enables column sorting by adding a sort key constraint to an existing table. See Sorted Tables

These SQL DML commands are supported:

Command

Description

INSERT INTO <table-full-name> VALUES (col1-value, col2-value, …)

Inserts a row; all columns must be provided in table order

INSERT INTO <table-full-name> (col1-name, col2-name, …) VALUES (col1-value, col2-value, …)

Inserts a row with values for a specified subset of columns

INSERT INTO <table-full-name> SELECT …

Inserts rows produced by a SELECT query into the target table

UPDATE <table-full-name> SET col1-name = col1-value, col2-name = col2-value WHERE condition

Updates one or more column values in rows that match the specified condition

DELETE FROM <table-full-name> WHERE condition

Deletes rows from a table that match the specified condition

INSERT INTO <table-full-name>

(part_colname_1, partition_colname_2, partition_colname_3, partition_colname_4, $parquet_file_path")

VALUES (value1_1, value2_1, value3_1, parquet_file_path_1),

(value1_2, value2_2, value3_2, parquet_file_path_2),...

Inserts data from parquet files into a partitioned table, according to the partition columns

Supported Aggregation Functions

The VAST Query Engine supports these aggregation functions:

Function

Example

Expected result (short explanation)

any_value(x)

Note

The column, x, must be constant size.

SELECT any_value(country) FROM t;

Returns the first non‑NULL country encountered (order‑sensitive).

avg(x)

SELECT avg(salary) FROM t;

Average of non‑NULL salaries; NULLs are ignored; empty group → NULL.

mean(x)

SELECT mean(salary) FROM t;

Average of non‑NULL salaries; NULLs are ignored; empty group → NULL.

sum(x)

SELECT sum(amount) FROM t;

Sum of non‑NULL amounts. For BOOLEAN, counts TRUEs.

min(x)

Note

The column, x, must be constant size.

SELECT min(score) FROM t;

Minimum non‑NULL score.

max(x)

Note

The column, x, must be constant size.

SELECT max(score) FROM t;

Maximum non‑NULL score.

bool_and(x)

SELECT bool_and(ok) FROM t;

TRUE if all non‑NULL booleans are TRUE; FALSE otherwise.

bool_or(x)

SELECT bool_or(ok) FROM t;

TRUE if any non‑NULL boolean is TRUE; FALSE if none are TRUE.

count()

SELECT count() FROM t;

Row count (includes NULLs). Same as count(*).

count(x)

SELECT count(user_id) FROM t;

Count of non‑NULL user_id values.

countif(cond)

SELECT countif(score >= 90) FROM t;

Number of rows where condition is TRUE.

product(x)

SELECT product(multiplier) FROM t;

Product of all non‑NULL multipliers (empty group → NULL).

stddev_pop(x)

SELECT stddev_pop(v) FROM t;

Population standard deviation.

stddev_samp(x) / stddev(x)

SELECT stddev_samp(v) FROM t;

Sample standard deviation (Bessel correction).

var_pop(x)

SELECT var_pop(v) FROM t;

Population variance.

var_samp(x) / variance(x)

SELECT var_samp(v) FROM t;

Sample variance (Bessel correction).

approx_count_distinct(x)

SELECT approx_count_distinct(user_id) FROM t;

Approximate cardinality using HyperLogLog.

Accessing VAST DataBases using the Query Engine

In order to access the VAST DataBase using the Query Engine, you will need the following:

Creating Virtual IP Pools for the VAST Query Engine

The Query Engine requires a dedicated virtual IP pool for client access. Create this pool with the role Query Engine, using the VAST Web UI.

  1. Navigate to the Network Access page, and select Virtual IP Pools.

  2. Click Create Virtual IP Pool.

  3. Compete details for the virtual IP pool following the procedure described in Creating Virtual IP Pools, selecting role type Query Engine. Define also a FQDN in the DNS for the virtual IP pool (see Step 6).

VAST DataBase Query Engine ADBC Driver for Client Applications

VAST provides an open-source ADBC driver that you can use to provide access for your client application to the VAST Query Engine.

Download the driver from https://github.com/vast-data/vastdb-adbc-driver.

The ADBC can be used in a Linux x86_64 environment, for programming languages that have a driver manager (for example, C++, Rust, Python, Ruby, Go, and Java).

When using this driver, you will need to provide a virtual IP address and an access/secret key pair, to access databases (see Creating Virtual IP Pools for the VAST Query Engine.

ADBC User Impersonation

You can run VAST ADBC driver commands with the credentials of the end user (instead of using the service account of the VAST ADBC driver). To do so:

  • Grant the s3:EndUserImpersonation permission to the VAST ADBC driver's service account through the identity or bucket policy. For example:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "Allow end user impersonation on Tabular operations",
          "Effect": "Allow",
          "Action": "s3:EndUserImpersonation",
          "Resource":  ["arn:aws:s3:::*"]
        }
      ]
    }
  • Use the vast.db.end_user connection option to pass the username of the end user along with the query. For example:

    conn = adbc_driver_manager.dbapi.connect(
       driver="driver_path",
       db_kwargs={
           "vast.db.endpoint": "server_ip",
           "vast.db.access_key": "access_key",
           "vast.db.secret_key": "secret_key",
       },
       conn_kwargs={
           "vast.db.end_user": "user_name",
       }
    )