S3 Bucket Logging

Overview

S3 bucket logging, or S3 server access logging, is a native S3 feature that VAST Cluster supports on S3-enabled views.

With S3 bucket logging, you can set up detailed recording of S3 operations on your bucket or bucket objects, with the logs written to another bucket.

Note
Operations performed through other access protocols are not subject to S3 bucket logging.

When the source bucket has S3 bucket logging enabled, VAST Cluster creates a log entry in AWS log format for each S3 request made to the source bucket, and periodically uploads the log objects to a destination bucket. The format of log object keys can be configured to allow for date-based partitioning of log objects.

You can set up S3 bucket logging in VAST Web UI, VAST CLI or via VAST Cluster S3 API.

Permissions for accessing the log objects in the destination bucket can be managed via a bucket policy, an identity policy, or when sending a PutBucketLogging request to the VAST Cluster S3 server.

To manage retention of log objects, configure VAST Web UI lifecycle rules on the destination bucket.Overview of Lifecycle Rules

Destination Bucket Requirements

The destination bucket must meet the following requirements:

The destination bucket is on a view that has S3 protocol enabled.
The view policy of the destination bucket view is set to S3 Native security flavor.
The destination bucket has the same bucket owner as the source bucket.
The destination bucket is not the same as the source bucket.
The destination bucket does not have S3 object locking enabled.S3 Object Locking Overview

Configuring S3 Bucket Logging in VAST Web UI

You configure S3 bucket logging for a bucket when creating or editing a view that exposes the bucket to be logged (the source bucket).Creating ViewsModifying Views

In the Bucket Logging tab of the Create View or Update View dialog, make the following settings:

Enable bucket logging	Toggle on or off to enable or disable S3 bucket logging for the bucket. By default, bucket logging is disabled.
Destination bucket	Select the bucket to store the logs.
Prefix	Optionally, specify a prefix that will be prepended to each key of a log object uploaded to the destination bucket. This prefix can be used to categorize log objects; for example, if you use the same destination bucket for multiple source buckets. The prefix can be up to 128 characters and must follow S3 object naming rules.
Key format	Select the format for the log object keys: Non-date-based partitioning This is the default format: [DestinationPrefix][YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString] Date-based partitioning This format enables timestamp-based partitioning of log objects: [DestinationPrefix][SourceUsername]/[SourceBucket]/ [YYYY]/[MM]/[DD]/[YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString] If you choose this format, you can use the Timestamp field to determine which time to use for the log object key: the time when the log object has been delivered to the destination bucket, or the time when the logged events occurred. In the formats: `[DestinationPrefix]` is the optional prefix that prepends keys of log objects uploaded to the destination bucket. You define this prefix in the Prefix field. `[SourceUsername]` is the username for the owner of the bucket being logged. `[SourceBucket]` is the bucket being logged. UTC time is used in timestamps. `[UniqueString]` is a unique string added to prevent overwriting of objects.
Timestamp	If you specified the Key format that enables date-based partitioning of log objects, select the type of timestamp to be used when generating log object keys: S3 event time. The timestamp shows the time when the logged events occurred. Log object delivery time. The timestamp shows the time when the log object has been delivered to the destination bucket.

Configuring S3 Bucket Logging via VAST S3 API

VAST Cluster supports the following S3 requests that are specific to S3 bucket logging:

PutBucketLogging is used to configure S3 bucket logging for a bucket and set permissions for accessing the log objects in the destination bucket (see here for an example).
GetBucketLogging returns the S3 bucket logging configuration for the bucket (see here for an example).

The following users are able to configure S3 bucket logging and return logging configuration via VAST S3 API:

The bucket owner
Users allowed to do so by applicable identity or bucket policy

Setting Permissions for Log Objects in the Destination Bucket

Permissions for accessing the log objects in the destination bucket can be managed via a bucket policy, an identity policy, through a PutBucketLogging request, or based on the destination bucket ACL.

To set permissions through a PutBucketLogging request (see here for an example):

Ensure that the destination bucket has ACLs enabled. If ACLs are disabled for the destination bucket, a PutBucketLogging request that sets grantees for this bucket would fail.S3 Object Ownership
In the Grantee element of the request, specify grantees by EmailAddress.
In the Permission element of the request, specify the permissions provided to the grantee.
These permissions are mapped to bucket ACL entries as follows:
Permission in PutBucketLogging Request
Permission in Bucket ACL
FULL_CONTROL
FULL_CONTROL
READ
READ
WRITE
WRITE
READ_ACP
WRITE_ACP

Permission in PutBucketLogging Request	Permission in Bucket ACL
FULL_CONTROL	FULL_CONTROL
READ	READ
WRITE	WRITE
	READ_ACP
	WRITE_ACP

Note
Permissions specified in the PutBucketLogging request override permissions in the existing bucket ACL.

Bucket Log Format

The S3 bucket log is created in AWS log format. Each entry in the log includes the following fields:

Field	Description
Bucket owner	The owner of the bucket being logged, in the format `username@domain` or just `username` if domain information is not available.
Bucket	The name of the bucket being logged.
Time	The UTC timestamp of when the request was received, in `[%d/%b/%Y:%H:%M:%S %z]` format.
Remote IP	The IP address from which the request was received.
Requester	The user which sent the request, in the format `username@domain` or just `username` if domain information is not available.
Request ID	The ID set by the server for this request.
Operation	The requested operation in the `REST.<HTTP method>.<resource type>` format; for example, `REST.GET.TAGGING`.
Key	The name of the object specified in the request.
Request URI	The `Request-URI` part of the HTTP message.
HTTP status	The HTTP status code of the response.
Error code	`-` if no S3 error occurred; otherwise, the S3 error code.
Bytes sent	Response bytes sent, excluding HTTP protocol overhead. `-` means zero bytes.
Object size	The size (in bytes) of the object specified in the request
Total time	The time (in milliseconds) elapsed from when the server received the request to when it sent the last byte of the response.
Turn-around time	The time (in milliseconds) elapsed from when the server received the last byte of the request to when it sent the first byte of the response.
Referer	The value of the `Referer` header in the HTTP message.
User agent	The value of the `User-Agent` header in the HTTP message.
Version ID	The ID of the object version, if applicable.
Host ID	This field is set to be the same as the Request ID.
Signature version	The version of the signature used to authenticate the request (`SigV2` or `SigV4`).
Cipher suite	If HTTPS was used, this field specifies the negotiated SSL cipher. For HTTP, it is set to `-`.
Authentication type	Whether the request was authenticated using the HTTP `AuthHeader` header, or by means of `QueryString`.
Host header	The value of the `Host` header in the HTTP message.
TLS version	The TLS version used, for example, `TLSv1.2`.
Access point ARN	This field is set to `-`.
aclRequired	Whether ACLs were used to authorize access: `Yes` if ACLs were used, or `-` if ACLs were not used.

For example:

[myuser c-2024-09-12-111045-288000 [12/Sep/2024:11:18:21 +0000]
 198.51.100.2 myuser 88033944677064 REST.PUT.TAGGING 
elbencho-198.51.100.4-S65536-10485760/dir0/dir1/r0/d1/r0-f2
"PUT /c-2024-09-12-111045-288000/elbencho-198.51.100.4-S65536-10485760/dir0/dir1/r0/d1/r0-f2?tagging HTTP/1.1"
200 0 - 1589570 10933984 10880249 "-" 
"aws-sdk-cpp/1.11.335 ua/2.0 md/aws-crt#0.26.8 
os/Linux/3.10.0-957.27.2.el7.x86_64 md/arch#x86_64 lang/c++#C++17 
md/GCC#13.2.1 cfg/retry-mode#standard api/S3ocore/1.34.61esource"
0 88033944677064 SigV4 - AuthHeader 198.51.100.3:9090090 - - Yes]