Vector Database Index

A Vector Database table stores n-dimensional vectors that represent items in some domain. Proximity in the vector space typically indicates proximity in the domain. A vector database index is a metric of the vector elements, that allows searches for similar vectors to performed quickly, especially at scale. Given an item in the domain, searching for similar items in the domain can be done by creating the vector that represents the item (also referred to as the item’s vector “embedding”), searching for the vector’s nearest neighbors in the vector database index, and then mapping those vectors back to the items they represent. The vector Database index efficiently organizes the vectors such that this search is fast, accurate, and efficient with compute resources. This can dramatically improve performance for queries on large model databases.

Indexes are designated for vector database tables at the time the table is created. They cannot be designated afterwards. Once designated for a table, they cannot afterwards be removed.

Indexing can be applied to a single, specific column in the table, which is of type vector. When you designate the index, you also designate the type of metric that is used to measure nearest neighbors. This can be either Euclidean or Cosine inner product.

Note

The term 'vector' is now a reserved word, and cannot be used as a name of a column for queries using the VAST Query Engine.

See Configuring Vector Indexing.

This example illustrates how to create an indexed table using the VAST DB Python SDK.

columns = pa.schema([
    ('id', pa.int64()),
    ('emb', pa.list_(pa.field(name="item", type=pa.float32(), nullable=False), dim)),
])

with session.transaction() as tx:
    bucket = tx.bucket(bucket_name)
    schema = bucket.create_schema(schema_name)
    non_indexed_table = schema.create_table(f"{table_name}_non_indexed", columns)
    vector_index_spec = VectorIndexSpec(column=vector_column_name,distance_metric=metric)
    indexed_table = schema.create_table(table_name, columns, vector_index=vector_index_spec)

This example illustrates how to create an indexed table using the VAST Query Engine.

create_indexed_table_sql = f"""
CREATE TABLE {indexed_table_path} (
    id BIGINT,
    {vector_col_name} vector(float, {dim}),
    two_chars VARCHAR,
    tinyint TINYINT,
    four_chars VARCHAR
)
VECTOR CLUSTER BY {vector_col_name} (metric = '{metric_sql}');
"""
adbc_session.execute_with_autocommit(create_indexed_table_sql)