Kafka Protocol Support

The VAST implementation of the Kafka protocol supports a basic subset of the Kafka APIs to allow clients to publish and consume events from the VAST Event Broker.

• VAST Event Broker supports:

  • Producer API

  • Consumer API

  • Consumer groups

  • Database queries on topics

  • Admin API:

    • Create topics

    • Get topic configuration ( ​describeConfigs​​ )

    • Update topic configuration (​​alterConfigs​​)

    • Delete topics

    • Delete consumer groups

  • Topic compaction

  • SSL

The following Kafka capabilities are not supported:

• Over-the-wire compression of messages. Note that VAST compression of data is supported.

• Automatic creation of topics

• Transactions

The following limitations apply:

• Producer API:

  • Messages are limited to 1MB.

  • In the event record, the key is limited to 126KB and the value is limited to 126KB.

  • Idempotent producing is not supported.

  • Automatic creation of topics is not supported.

• Consumer API:

  • No more than 256 consumer groups per view (broker)

  • The following is not supported:

    • Consumer group stickiness parameters (such as ​group.instance.id​​)

    • READ UNCOMMITTED isolation level

    • Cooperative rebalancing

    • Client rack awareness

    • Fetch sessions (only full fetch will be applied), delayed fetch parameters

    • Seek by time

• Only one virtual IP pool can be associated with a Kafka-enabled view.

• The amount of VAST Event Broker views that you can create on a VAST cluster, is limited by the maximum number of views supported by the cluster and by the maximum number of virtual IP pools (since each broker view requires a dedicated virtual IP pool). See ​VAST Cluster Scale Guidelines​​ for details.

• The amount of event topics that you can create on a VAST cluster, is limited by the maximum number of tables per VAST Database table (see ​VAST Cluster Scale Guidelines​​) and the overall amount of event topic partitions.

• A topic can have up to 20,000 partitions. The number of partitions in a topic cannot be changed after the topic has been created. Up to 200,000 partitions are supported per VAST Event Broker view.

• Event queries based on the topic partition are not supported.

• When listing consumer groups, the response is limited to 256 groups per Kafka-enabled view.

• VAST replication of consumer groups is not supported.

• Event publishing and consuming operations, as well as topic management operations are not subject to VAST Protocol Auditing or Quality of Service (QoS).

Supported Kafka Clients

VAST Cluster supports Confluent Kafka Python client 2.4 - 2.8.