VAST Event Broker collects events occurring to elements in VAST Element Store into Kafka topics. VAST Event Broker is based on the VAST Database which allows consumers to query Kafka topics via the various database APIs.
VAST Event Broker utilizes VAST proprietary implementation of the Kafka protocol, as detailed in Kafka Protocol Support.
The workflow is as follows:
Kafka Protocol Support
The VAST implementation of the Kafka protocol supports a basic subset of the Kafka APIs to allow clients to publish and consume events from the VAST Event Broker.
VAST Event Broker supports:
Producer API
Consumer API
Consumer groups
Database queries on topics
Admin API (create topics, delete topics and consumer groups)
The following Kafka capabilities are not supported:
Over-the-wire compression of messages
Tip
VAST compression of data is supported.
Topic compaction
Automatic creation of topics
Authentication and authorization
SSL
Transactions
The following limitations apply:
Producer API:
Messages are limited to 1MB.
In the event record, the key is limited to 126KB and the value is limited to 126KB.
Access to topics by UUID is not supported.
Idempotent producing is not supported.
Automatic creation of topics is not supported.
Consumer API:
No more than 256 consumer groups per view (broker)
The following is not supported:
Consumer group stickiness parameters (such as
group.instance.id)READ UNCOMMITTED isolation level
Cooperative rebalancing
Client rack awareness
Fetch sessions (only full fetch will be applied), delayed fetch parameters
Seek by time
Admin API:
Supported APIs include the APIs to create topics, delete topics, and to delete groups.
Only the following topic parameters are supported:
Number of partitions
Topic retention period
Only one virtual IP pool can be associated with a Kafka-enabled view.
A topic can have up to 1000 partitions. The number of partitions in a topic cannot be changed after the topic has been created.
Event queries based on the topic partition are not supported.
When listing consumer groups, the response is limited to 256 groups per Kafka-enabled view.
VAST replication of consumer groups is not supported.
Event publishing and consuming operations, as well as topic management operations are not subject to VAST Protocol Auditing or Quality of Service (QoS).
Configuring a View for VAST Event Broker
The view that exposes elements for which you want to publish events must be configured as follows:
Specify Kafka as the view's protocol:
In VAST Web UI, select Kafka in the Protocols field in general view settings (Element Store -> Views -> choose to create a view -> General tab).
In VAST CLI, run the
view createcommand with the--protocols KAFKAoption specified.
Note
Adding the Kafka protocol to an existing view is not allowed.
Specify the virtual IP pool to be used to access the event topics using the Kafka protocol. The pool must belong to the same VAST tenant as the Kafka-enabled view. The pool must have enough virtual IPs so that there is at least one virtual IP per CNode.
If the view is associated with a view policy that includes virtual IP pools, the pool specified as the Kafka pool must be one of the view policy pools.
In VAST Web UI, select a pool in the VIP Pool field in view settings (Element Store -> Views -> choose to create a view -> Kafka tab).
In VAST CLI, run the
view createcommand with the--kafka-vip-pools <pool ID>option specified.
The following limitations apply to Kafka-enabled views:
Only one virtual IP pool can be associated with a Kafka-enabled view.
Once the view has been created, the virtual IP pool cannot be replaced by another one (but it can be modified if needed).
View nesting is not supported.
Managing Event Topics
You create and manage event topics similar to VAST Database tables. When you create a Kafka-enabled view, VAST Cluster automatically creates an associated VAST database that contains a schema named kafka_topics. Under this schema, you can create topics where events will be recorded.
Event topics can be queried via the VAST DB API but cannot be modified.
Creating a Topic in VAST Web UI
To create a topic in VAST Web UI:
In the left navigation menu, choose DataBase -> VAST Database to open a list of databases available on the cluster.
Click an arrow on the left to the database named after your Kafka-enabled view and then click Kafka-Compatible Broker Topics under the database name.
Click the Add Topic button on the right to create a new topic.
In the Create Topic dialog, complete the following required fields:
Topic Name
Enter a name for the topic.
Number of partitions
Enter the number of partitions for the topic.
Each partition can hold up to 1000 topics. The number of partitions in a topic cannot be changed after the topic has been created.
Retention period
Specify the amount of time to keep an event record in the topic. When the retention period for a record expires, the records is deleted from the topic. The default retention period is seven days. The minimum allowed retention period is 6 hours.
Click Create.
Managing Topics in VAST CLI
Use the following VAST CLI commands to manage event topics:
Task | Command |
|---|---|
Create a topic | |
Modify a topic | |
Delete a topic | |
List topics | |
View a topic |
Querying Events from VAST Event Broker Topics
VAST Event Broker topics can be queried through the VAST Database API, and also using VAST Database connectors (e.g. the connectors for Trino and Spark).
Monitoring Event Traffic with VAST Analytics
VAST Cluster provides analytic reports that show latency and bandwidth for produced and fetched events and messages per CNode and per view.
To access Kafka-related analytics in VAST Web UI, go to Analytics -> Analytics and in the Predefined Analytics tab, search for reports containing Kafka in their name.