Publishing Events to Third-Party Event Brokers

  • Updated on Jan 30, 2026
  • Published on Jan 28, 2026
  • 6 minute(s) read
Prev Next

VAST Cluster supports Apache Kafka as the event broker. Event publishing is only available for S3 Bucket views.  

The workflow is as follows: 

  1. Add a third-party event broker to the VAST cluster.

  2. Set up S3 event notifications for the view for which you want to publish events. 

  3. Verify the configuration by sending a test event to the broker.

  4. Monitor event publishing performance on the VAST cluster.

For information about S3 event record format, see S3 Event Record Format.

Managing Third-Party Event Brokers

Managing Third-Party Event Brokers in VAST Web UI

You can view, add, edit and delete event brokers in the External Kafka Brokers tab in cluster notification settings (Settings -> Notifications -> External Kafka Brokers).

To add an event broker to the cluster:

  1. Go to Settings -> Notifications -> External Kafka Brokers.

  2. Click Add a New External Kafka Broker and complete these fields:

    Name

    Enter a name for the event broker configuration.

    Host

    Enter the bootstrap URL of the event broker server. You can specify an IP or FQDN. 

    If the Kafka cluster runs multiple event brokers, click +Add to add more hosts. You can add up to five hosts.

    Port

    Enter a port to communicate with the event broker server.

    Tip

    Ensure that the hosts are accessible from the VAST cluster's management interface at the specified ports.

  3. Click Add External Kafka Broker. 

    The newly created event broker configuration is added to the list of event brokers.

To edit or remove an event broker, right-click it and choose Edit or Remove.

Configuring S3 Bucket Notifications for a View

When creating a new event notification definition, ensure that it does not overlap with any existing definitions. Per the AWS standard, two definitions cannot potentially publish events for the same S3 operation. For example, you cannot have a definition that publishes an event on creation of an object with the abcdef object key prefix and a definition for creation of objects prefixed with abc.

Configuring S3 Bucket Notifications for a View in VAST Web UI

To configure event publishing for a view, use the Bucket Notifications tab in view settings (Element Store -> Views -> choose to create or edit a view). 

Note

Event publishing settings are only available if the view has the S3 Bucket protocol enabled and is controlled with the S3 Native security flavor.

You can specify the following:

  • One or more types of events for which you want to send notifications.

  • The event broker (from those configured on the VAST cluster) and the topic to which to publish events. 

  • (Optionally) Object key prefix and/or suffix to have the events filtered based on S3 object keys.

Note

For a complete procedure, see Creating Views.

Verifying Event Publishing Configuration in Kafka Broker

After you've saved an event notification definition for a view, VAST Cluster sends a test event message to the associated Kafka broker to ensure that the setup works as expected.

The test event message is compatible to the AWS bucket notification test event format, for example:

{  
   "Service":"Vast S3",
   "Event":"s3:TestEvent",
   "Time":"<timestamp>",
   "Bucket":"<bucket name>",
   "RequestId":"<request ID>",
   "HostId":"<host ID>"
}

To verify that your event publishing configuration is working as expected:

  • Check if the test event message has reached the broker from within the Kafka client, by looking at the messages under the topic you specified in the event notification definition in VAST Cluster.

  • Ensure that the intended event consumer application(s) is able to receive the test event without any issues.

Monitoring Event Publishing to Third-Party Event Brokers on VAST Cluster

Use the following predefined analytic reports to monitor metrics related to event publishing (Analytics -> Analytics -> Predefined -> View):

  • Published Events Count shows the accumulated number of published events.

  • Published Events Latency shows the following metrics:

    • Create event latency refers to the amount of time needed to generate an event notification within the VAST cluster.

    • Send event latency refers to the amount of time needed to send the event notification to the event broker.

S3 Event Record Format

VAST Cluster generates S3 event records in the following format:

   "Records":[  
      {  
         "awsRegion"="<region>",  
         "eventName":"<event type>",  
         "eventSource":"vast:s3",   
         "eventTime":"<timestamp>"  
         "eventVersion":"2.2",
         "requestParameters":{  
            "sourceIPAddress":"<source IP>"
         },
         "responseElements":{  
            "x-amz-request-id":"<request ID>",
            "x-amz-id-2":"<request ID>"
         },
         "s3":{  
            "bucket":{  
               "arn":"arn:vast:s3:::<bucket name>"
               "name":"<bucket name>",
               "ownerIdentity":{  
                  "principalId":"<principal ID>"
               },
            },
            "configurationId":"<config ID>",
            "object":{  
               "eTag":"<eTag>",               
               "key":"<key>",
               "versionId":"<version>",
               "sequencer": "<sequencer>"
               "size":"<size>",
            }
            "s3SchemaVersion":"1.0",
         },
         "userIdentity":{  
            "principalId":"<principal's email>"
         },
      }
   ]

Where:

  • <timestamp> contains the time when VAST Cluster finished processing of the request. The time is in ISO-8601 format, for example, 1970-01-01T00:00:00.00000Z.

  • <event type> identifies the S3 event type (including the prepending 's3:').

  • <principal ID> identifies the principal (for example, the username for local users, or username@domainname for users retrieved from authentication providers).

  • <source IP> is the client IP address from which the S3 request originated.

  • <request ID> is the VAST-internal identifier for the request.

  • <config ID> is the ID of the event notification definition record created on the VAST cluster.

  • <bucket name> is the name of the bucket included in the S3 request.

  • <key> is the key of the object specified in the S3 request.

  • <size> indicates the object size in bytes.

  • <eTag> is the object's entity tag.

  • <version> indicates the object version if S3 Object Versioning is enabled for the bucket. If versioning is not enabled, this entry is not included.

  • <sequencer> is a string representation of a hexadecimal value used to determine the event's sequence number. It is only used with PUT and DELETE requests.

For example:

Records=[Bunch(
    awsRegion='region',
    eventName='s3:ObjectCreated:Put',
    eventSource='vast:s3',
    eventTime='2024-10-01T10:51:35.859365Z',
    eventVersion='2.2',
    requestParameters=Bunch(sourceIPAddress='198.51.100.2'),
    responseElements=Bunch(x-amz-id-2='0x80010000000c', x-amz-request-id='0x80010000000c'),
    s3=Bunch(
         bucket=Bunch(
              arn='vast:s3:::design-enter-above',
              name='design-enter-above',
              ownerIdentity=Bunch(principalId='user@example.comlab')
              ),
         configurationId='participant-full',
         object=Bunch(
              eTag='5defbfc99890b0538929cca518a1fe7f',
              key='drive-blue-capital',
              sequencer='000300000000000f4241',
              size=1048576
              ),
         s3SchemaVersion='1.0'
         ),
    userIdentity=Bunch(principalId='user@example.com')
)]
Related articles