Documentation Index

Fetch the complete documentation index at: https://kb.vastdata.com/llms.txt

Use this file to discover all available pages before exploring further.

blobexpansion create

Prev Next

This command configures parsing for Kafka events. Fields in the the event message are specified for parsing, as well as a database table in which the parsed message information is saved. The table must already exist. Columns are created in the target table for the fields that are parsed, if they do not exist already.

Usage

vcli: admin> blobexpansion create --database-name DATABASE 
                                  --table-name TABLE
                                  --target-table-name TABLE 
                                  --arrow-schema ARROW_SCHEMA
                                 [--tenant-id  TENANT]
                                 [--target-table-schema TARGET_SCHEMA]
                                 [--expansion-format EXPANSION_FORMAT]
                                 [--source-column-name COLUMN]
                                 [--copy-source-column]
                                 [--flatten-path]
                                 [--flatten-delimiter FLATTEN_DELIMITER]

Required Parameters

--database-name DATABASE

Kafka bucket name containing the source topic.

--table-name TABLE

Source Kafka topic name.

--target-table-name TABLE

Target database table name for expanded data.

--arrow-schema ARROW_SCHEMA

Column definitions as a comma-separate list of name-type pairs. These are the fields (and their type) in the Kafka event source that are parsed. They are parsed into columns of the same name and type in the target table. Columns are created in the target database table if they do not already exist.

Example --arrow-schema id-int64,status-string,score-double

This selects fields id, status, and score, for parsing from event messages.

Options

--tenant-id TENANT

Tenant ID (optional) for the schema.

--source-column-name COLUMN

Source column name (only 'value' is supported)

--target-table-schema TARGET_SCHEMA

Target table schema (optional)

--expansion-format EXPANSION_FORMAT

The format of the source column, that is expanded (only 'json' is supported)

--copy-source-column

If set, the raw source event message (in binary JSON format) is copied to the target table.

--flatten-path

Flatten nested JSON paths. If set, nested JSON paths are parsed as a flat path.

Example: this JSON block:

"user": {
    "profile": {
      "name": "Alice",
      "age": 30
    }
  }

is parsed to user_profile_name, user_profile_age (where the delimiter is '_').

--flatten-delimiter FLATTEN_DELIMITER

Delimiter for flattened paths (default: '__')

Example

This example shows the blobexpansion create command with sample values.

vcli: admin> blobexpansion create --tenant-id 1 --database-name kafka_bucket --table-name my_topic --source-column-name value --target-table-name expanded_data --target-table-schema public --expansion-format json --arrow-schema col1-string,col2-int,col3-float --copy-source-column --flatten-path --flatten-delimiter __