Event Publishing

  • Updated on Feb 2, 2026
  • Published on Jan 27, 2026
  • 7 minute(s) read
Prev Next

Overview

VAST Cluster can publish events to event brokers.

In the context of event publishing, an event is a change that has occurred for an element in VAST Element Store, such as creation of a new object in an S3 bucket. When a certain event is encountered, VAST Cluster can send a JSON record of it to the connected event broker, where the information can be consumed by applications. 

As of version 5.2, VAST Cluster supports Apache Kafka as the event broker. Event publishing is only available for S3 bucket views.

VAST Cluster generates event records for S3 requests that were completed successfully. Event records for failed S3 requests are not created.

To take advantage of event publishing: 

  1. Add an event broker to the VAST cluster.

  2. Set up S3 event notifications for an S3 bucket view for which you want to publish events. 

To view and manage Kafka event brokers, use either of the following: 

  • The External Kafka Brokers tab in cluster notification settings (Settings -> Notifications -> External Kafka Brokers) in VAST Web UI

  • Kafka broker management commands (kafkabroker <action>) in VAST CLI

  • The /kafkabroker/ and /kafkabroker/<ID>/ endpoints of VAST REST API.

To configure S3 event notifications for a view, use any of the following:

  • The Bucket Notifications tab in view settings (Element Store -> Views -> choose to create or edit a view) in the VAST Web UI.

  • Event notification management commands (eventnotification <action>) in the VAST CLI.

  • The event_notifications parameter for the /views/ and /views/<ID> endpoints of VAST REST API.

  • The GetBucketNotificationConfiguration and PutBucketNotificationConfiguration actions of the S3 API.

Note

When creating a new event notification definition, ensure that it does not overlap with any existing definitions. Per the AWS standard, two definitions cannot potentially publish events for the same S3 operation. For example, you cannot have a definition that publishes an event on creation of an object with the abcdef object key prefix and a definition for creation of objects prefixed with abc.

Managing Event Brokers in VAST Web UI

You can view, add, edit and delete event brokers in the External Kafka Brokers tab in cluster notification settings (Settings -> Notifications -> External Kafka Brokers).

To add an event broker to the cluster:

  1. Go to Settings -> Notifications -> External Kafka Brokers.

  2. Click Add a New External Kafka Broker and complete these fields:

    Name 

    Enter a name for the event broker configuration.

    Tenant 

    Select one tenant for which the event broker will be available, or leave the default of All tenants so that the broker is available for all tenants.

    Host 

    Enter the bootstrap URL of the event broker server. You can specify an IP or FQDN. 

    If the Kafka cluster runs multiple event brokers, click +Add to add more hosts. You can add up to five hosts.

    Port 

    Enter a port to communicate with the event broker server.

    Tip

    Ensure that the hosts are accessible from the VAST cluster's management interface at the specified ports.

  3. Click Add External Kafka Broker. 

    The newly created event broker configuration is added to the list of event brokers.

To edit or remove an event broker, right-click it and choose Edit or Remove.

Managing Event Brokers in VAST CLI

Use the following VAST CLI commands to manage event brokers:

Task

Command

Add an event broker

kafkabroker create 

Modify an event broker

kafkabroker modify 

Remove an event broker

kafkabroker delete

List event brokers

kafkabroker list

View details of an event broker

kafkabroker show

List topics on an event broker

kafkabroker list-topics

Configuring Event Publishing for a View in VAST Web UI

To configure event publishing for a view, use the Bucket Notifications tab in view settings (Element Store -> Views -> choose to create or edit a view). 

Note

Event publishing settings are only available if the view has the S3 Bucket protocol enabled and is controlled with the S3 Native security flavor.

You can specify the following:

  • One or more types of events for which you want to send notifications.

  • The event broker (from those configured on the VAST cluster) and the topic to which to publish events. 

  • (Optionally) Object key prefix and/or suffix to have the events filtered based on S3 object keys.

Note

For a complete procedure, see Creating Views.

Configuring Event Publishing for a View in VAST CLI

Use the following VAST CLI commands to configure event publishing:

Task

Command

Create an event notification 

eventnotification create 

Modify an event notification 

eventnotification modify

Delete an event notification

eventnotification delete

List event notifications

eventnotification list

View event notification details

eventnotification show

Verifying Event Publishing Configuration in Kafka Broker

After you've saved an event notification definition for a view, VAST Cluster sends a test event message to the associated Kafka broker to ensure that the setup works as expected.

The test event message is compatible to the AWS bucket notification test event format, for example:

{  
   "Service":"Vast S3",
   "Event":"s3:TestEvent",
   "Time":"<timestamp>",
   "Bucket":"<bucket name>",
   "RequestId":"<request ID>",
   "HostId":"<host ID>"
}

To verify that your event publishing configuration is working as expected:

  • Check if the test event message has reached the broker from within the Kafka client, by looking at the messages under the topic you specified in the event notification definition in VAST Cluster.

  • Ensure that the intended event consumer application(s) is able to receive the test event without any issues.

Monitoring Event Publishing on VAST Cluster

Use the following predefined analytic reports to monitor metrics related to event publishing (Analytics -> Analytics -> Predefined -> View):

  • Published Events Count shows the accumulated number of published events.

  • Published Events Latency shows the following metrics:

    • Create event latency refers to the amount of time needed to generate an event notification within the VAST cluster.

    • Send event latency refers to the amount of time needed to send the event notification to the event broker.

Supported S3 Events

VAST Cluster supports a subset of S3 events:

  • Object creation events:

    Event

    Description

    s3:ObjectCreated:Put 

    reation of an S3 object by a PUT request

    s3:ObjectCreated:Post 

    Creation of an S3 object by a POST request.

    s3:ObjectCreated:Copy 

    Creation of an S3 object by copying an existing object

    s3:ObjectCreated:CompleteMultipartUpload 

    Creation of an S3 object as a result of a multi-part upload

    s3:ObjectCreated:* 

    Any S3 object creation.

  • Object deletion events:

    Event

    Description

    s3:ObjectRemoved:Delete 

    Deletion of an S3 object by the S3 DELETE operation.

    s3:ObjectRemoved:DeleteMarkerCreated 

    Creation of a delete marker for a versioned S3 object.

    s3:ObjectRemoved:* 

    Any S3 object deletion.

S3 Event Record Format

VAST Cluster generates S3 event records in the following format:

   "Records":[  
      {  
         "awsRegion"="<region>",  
         "eventName":"<event type>",  
         "eventSource":"vast:s3",   
         "eventTime":"<timestamp>"  
         "eventVersion":"2.2",
         "requestParameters":{  
            "sourceIPAddress":"<source IP>"
         },
         "responseElements":{  
            "x-amz-request-id":"<request ID>",
            "x-amz-id-2":"<request ID>"
         },
         "s3":{  
            "bucket":{  
               "arn":"arn:vast:s3:::<bucket name>"
               "name":"<bucket name>",
               "ownerIdentity":{  
                  "principalId":"<principal ID>"
               },
            },
            "configurationId":"<config ID>",
            "object":{  
               "eTag":"<eTag>",               
               "key":"<key>",
               "versionId":"<version>",
               "sequencer": "<sequencer>"
               "size":"<size>",
            }
            "s3SchemaVersion":"1.0",
         },
         "userIdentity":{  
            "principalId":"<principal's email>"
         },
      }
   ]

Where:

  • <timestamp> contains the time when VAST Cluster finished processing of the request. The time is in ISO-8601 format, for example, 1970-01-01T00:00:00.00000Z.

  • <event type> identifies the S3 event type (including the prepending 's3:').

  • <principal ID> identifies the principal (for example, the username for local users, or username@domainname for users retrieved from authentication providers).

  • <source IP> is the client IP address from which the S3 request originated.

  • <request ID> is the VAST-internal identifier for the request.

  • <config ID> is the ID of the event notification definition record created on the VAST cluster.

  • <bucket name> is the name of the bucket included in the S3 request.

  • <key> is the key of the object specified in the S3 request.

  • <size> indicates the object size in bytes.

  • eTag is the object's entity tag (eTag).

  • <version> indicates the object version if S3 Object Versioning is enabled for the bucket. If versioning is not enabled, this entry is not included.

  • <sequencer> is a string representation of a hexadecimal value used to determine the event's sequence number. It is only used with PUT and DELETE requests,

For example:

Records=[Bunch(
    awsRegion='region',
    eventName='s3:ObjectCreated:Put',
    eventSource='vast:s3',
    eventTime='2024-10-01T10:51:35.859365Z',
    eventVersion='2.2',
    requestParameters=Bunch(sourceIPAddress='198.51.100.2'),
    responseElements=Bunch(x-amz-id-2='0x80010000000c', x-amz-request-id='0x80010000000c'),
    s3=Bunch(
         bucket=Bunch(
              arn='vast:s3:::design-enter-above',
              name='design-enter-above',
              ownerIdentity=Bunch(principalId='user@example.comlab')
              ),
         configurationId='participant-full',
         object=Bunch(
              eTag='5defbfc99890b0538929cca518a1fe7f',
              key='drive-blue-capital',
              sequencer='000300000000000f4241',
              size=1048576
              ),
         s3SchemaVersion='1.0'
         ),
    userIdentity=Bunch(principalId='user@example.com')
)]

Related articles