S3cmd

Using S3cmd with VAST Cluster S3

Note
Documentation for S3cmd can be found on s3cmd's website. Please use the S3cmd documentation and inline command help as needed in conjunction with our specific guidelines for using S3cmd with VAST Cluster.

S3cmd is a command line client for uploading, retrieving and managing data with the S3 service.

The sections below describe how to install and configure S3cmd for use with VAST Cluster and how to use it to perform supported tasks.

Install and Configure S3cmd

Browse to https://s3tools.org/s3cmd for system requirements and download.

Make sure the s3cmd is installed on your client machine. For example, on CentOS/Redhat Linux systems:

sudo yum install s3cmd -y

Once s3cmd is installed, you need to configure it to connect to your VAST Cluster S3 account.

Configurations, including user account credentials, are specified in a configuration file. The default configuration file is located under your home directory ($HOME/.s3cfg). This file contains all of the S3cmd settings. All s3cmd commands default to the default configuration file.

You can create an additional configuration file for your VAST-specific settings, or add them to the default configuration file. When using an additional configuration file, specify the file in your S3cmd commands.

Minimal Configuration File Content

Your configuration file should minimally include the following content. Save it to your home directory as <filename>.s3cfg, where <filename> is a name of your choice. In our examples. we will use vast.s3cfg.

[default]
access_key = <ACCESS-KEY>
secret_key = <SECRET-KEY>
host_base = <CLUSTER VIRTUAL IP>
host_bucket = <CLUSTER VIRTUAL IP>
use_https = false|true

In which:

<ACCESS-KEY> and <SECRET-KEY> are your S3 key pair.
<CLUSTER VIRTUAL IP> can be any of the cluster's Virtual IPs.
Note
To retrieve the cluster's virtual lPs:
- In the VAST Web UI, from the left navigation menu choose Network Access -> Virtual IPs to open the Virtual IPs page. It shows you which virtual IPs are configured on each CNode.
- In the VAST CLI, run the vip list command.
Caution
Do not specify a port number. The port is not configurable. Specifying the default port can cause an error with signature version 4.
To connect over HTTPS, set use_https=true. When this is set, the service will listen on port 443, the secure port. HTTPS connection will only work if an SSL certificate for S3 is installed on the VMS. See Enabling an HTTPS Connection.
To connect over HTTP, set use_https=false.

Additional Configuration Options

For more configuration options, see s3cmd documentation. Configuration options can be passed in individual commands or in the configuration file.

For HTTPS Connections

If you are configuring S3cmd to connect to S3 over HTTPS, you need to make sure that the connection doesn't fail due to failed certificate verification. An SSL certificate must be installed on the VMS for the S3 service. The S3 service provides the installed certificate to S3cmd as part of the SSL handshake when establishing an HTTPS connection to the service. By default, S3cmd verifies the certificate using the default local certificate trust store. The local path to the default certificate store is OS dependent. If the default trust store doesn't recognize the CA that signed the certificate, such as with a self signed certificate, the connection will fail with the default configuration.

The following are commands you might include in the configuration file to handle SSL certificate verification:

To disable certificate verification:
```
check_ssl_certificate = False
```
Alternatively, to set a non-default path to the certificate trust store and point it to your certificate:
```
ca-certs = <CA_CERTS_FILE>
```
Often there is a mismatch between the hostname embedded in the certificate and the service endpoint URL, which causes failure when the hostname is verified. To disable checking the hostname:
```
no-check-hostname
```

Signature Versions

VAST Cluster supports both signature version 4 (default and recommended) and signature version 2. To enforce signature version 2 specify the following in the configuration file:

signature_v2=true

Create a Bucket

s3cmd [-c CONFIGFILE] mb s3://BUCKET [--ca-certs=CERTIFICATE]

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	Name of bucket
CERTIFICATE	SSL certificate for HTTPS connections.

Example

Create a bucket named bucket1 specifying our configuration file as the non default vast.s3cfg.

$ s3cmd -c vast.s3cfg mb s3://bucket1
Bucket 's3://bucket1/' created

For HTTPS:

$ s3cmd mb s3://testbucket --ca-certs=client_cert.crt
Bucket 's3://testbucket/' created

List All Buckets on the Server

s3cmd [-c CONFIGFILE] ls s3://

CONFIGFILE

Configuration file. Defaults to $HOME/.s3cfg

Example

List all buckets on the server, specifying the root path of all buckets (s3://) and specifying our configuration file as the non default vast.s3cfg.

$ s3cmd -c vast.s3cfg ls s3://
2019-06-24 09:55  s3://bucket1

List Objects in a Bucket

s3cmd [-c CONFIGFILE] ls [s3://BUCKET[/PREFIX]]}

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	Name of bucket
PREFIX	Filter objects by prefix. Although the S3 structure is flat consisting of buckets containing objects with no hierarchy of subbuckets and subfolders, object keys can include prefixes separated by a delimiter. Hierarchy can be inferred from the prefixes.

Example

List objects in the bucket bucket1.

$ s3cmd -c vast.s3cfg ls s3://bucket1
2019-06-24 19:51  205 s3://bucket1/blue.txt
2019-06-24 19:52  93  s3://bucket1/brown.txt
2019-06-24 19:40  176 s3://bucket1/vast.s3cfg

Retrieve Bucket Metadata

$ s3cmd [-c CONFIGFILE]-c info s3://BUCKET

CONFIGFILE	The name of your VAST configuration file
BUCKET	The name of the bucket

Example

Retrieve the metadata of the bucket bucket1.

$ s3cmd -c vast.s3cfg info s3://bucket1
s3://bucket1/ (bucket):
   Location:  vast-1
   Payer:     none
   Expiration Rule: none
   Policy:    none
   CORS:      none
   ACL:       vastuser: FULL_CONTROL

Delete a Bucket

s3cmd [-c CONFIGFILE] rb s3://BUCKET

Note
This command can be used with the --recursive flag which removes the bucket with all of its contents. This form of the command calls the DeleteObjects AWS S3 API which is supported from VAST Cluster 3.2.0.

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	Name of bucket

Example

Delete bucket bucket1.

$ s3cmd -c vast.s3cfg rb s3://bucket1

Set Bucket ACL

Before setting ACL permissions, we recommend you read Managing S3 Access Control Lists (ACLs).

s3cmd [-c CONFIGFILE] setacl s3://BUCKET[/OBJECT] [--acl-grant=PERMISSION:{USER|GROUP}] [--acl-revoke=PERMISSION:{USER_VAST_ID|GROUP}]

CONFIGFILE	Configuration file. Defaults to $HOME/s3cf
BUCKET	Name of bucket
--acl-grant	Grant stated permission for stated user or group.
--acl-revoke	Revoke stated permission for stated user or group
PERMISSION	Type of permission to grant: read write read_acp write_acp full_control all
USER	A user to which you want to grant the permission. Can be specified as: . A principal name in the format user@domain, where user is the user name and domain is configured for an external auth provider on the cluster (LDAP, NIS). Note Users on the VAST provider cannot be specified this way. A VID, which is a VAST ID used in the cluster's internal user database. A user VID is retrievable by running the user query VAST CLI command and specify udb as the context of the query. The output includes the user's VID.
GROUP	A group to which you want to grant the permission. The group can be specified as follows: Set the type to GroupLoginName and specify the group in the format group@domain, where group is the group name and domain is configured for an external auth provider on the cluster (such as LDAP). Specify the group's VID, which is the VAST ID used in the cluster's internal user database. A group VID can be retrieved by running the group query VAST CLI command and specify udb as the context of the query. The output includes the group's VID. If you want to grant permissions to an S3 predefined group, specify one of the following: `http://acs.amazonaws.com/groups/global/AllUsers`, or `http://acs.amazonaws.com/groups/global/AuthenticatedUsers`

Examples

Grant read permissions for bucket mybbucket to group mygroup@domain.com:

$ s3cmd setacl s3://mybucket --acl-grant=read:'groupLoginName=mygroup@domain.com'

Grant full control to the predefined group AuthenticatedUsers to access the bucket mybucket.

$ s3cmd setacl s3://mybucket --acl-grant='full_control:http://acs.amazonaws.com/groups/global/AuthenticatedUsers'

Grant read permission to the predefined AllUsers group to access the bucket mybucket.

$ s3cmd setacl s3://mybucket --acl-grant='read:http://acs.amazonaws.com/groups/global/AllUsers'

Grant user with VAST ID 14 write permission to the bucket mybucket:

$ s3cmd setacl s3://mybucket --acl-grant='write:14

Return Bucket ACL

s3cmd [-c CONFIGFILE] info s3://BUCKET

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	Name of bucket

Example

Retrieve all metadata of the bucket bucket1. The output includes the bucket's ACL.

$ s3cmd -c vast.s3cfg info s3://bucket1
s3://bucket1/ (bucket):
   Location:  vast-1
   Payer:     none
   Expiration Rule: none
   Policy:    none
   CORS:      none
   ACL:       vastuser: FULL_CONTROL

Upload an Object

s3cmd [-c CONFIGFILE] [--disable-multipart] put FILE [FILE...] s3://BUCKET[/PREFIX]

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
--disable-multipart	Disables multipart upload for files over 15 MB (which by default are loaded as multipart uploads)
FILE	Name of local file to upload
BUCKET	Name of bucket to which to upload file
PREFIX	Prefix to append to the object name from which file system hierarchy can be inferred. When listing objects in the bucket, the prefix can be used to filter the list. You can specify a nested series of prefixes delimited by the delimiter (which is typically '/'). Object listing can be filtered by prefixes and delimiters. See List Objects in a Bucket. For example, when uploading a file called file.asp, if you specify a prefix /dev/misc the complete S3 object key becomes dev/misc/file.asp. A file system hierarchy can be inferred consisting of a folder called misc nested under a folder called dev in which the file file.asp is contained.

Example

Add file blue.txt to bucket bucket1.

$ s3cmd -c vast.s3cfg put blue.txt s3://bucket1
upload: 'blue.txt' -> 's3://bucket1/blue.txt'  [1 of 1]
205 of 205   100% in  0s  4.33 kB/s  done

Create a Copy of an Object

s3cmd [-c CONFIGFILE] cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET1	Bucket from which to copy object
OBJECT1	Source object to copy
BUCKET2	Bucket in which to create new object
OBJECT2	Name of destination object, if different from source object

Example

Create copy of file.zip, which is stored in bucket mybucket, store the copy in the same bucket and call it copyoffile.zip:

$ s3cmd -c vast.s3cfg cp s3://mybucket/file.zip s3://mybucket/copyoffile.zip

Download an Object

s3cmd [-c CONFIGFILE] get s3://BUCKET/OBJECT LOCAL_FILE

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	Bucket from which to retrieve stored object
OBJECT	Object to download
LOCAL_FILE	The local file path to download to

Example

Download the file file.zip from the bucket bucket.zip.

$ s3cmd -c vast.s3cfg get s3://mybucket/file.zip file.zip

Retrieve Object Metadata

  s3cmd [-c CONFIGFILE] info s3://BUCKET[/OBJECT]

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	Bucket in which object is stored
OBJECT	Object for which to retrieve metadata

Example

Retrieve metadata of file vast.s3cfg.

$ s3cmd -c vast.s3cfg info s3://bucket1/vast.s3cfg
s3://bucket1/vast.s3cfg (object):
   File size: 176
   Last mod:  Mon, 24 Jun 2019 19:40:27 GMT
   MIME type: text/plain
   Storage:   STANDARD
   MD5 sum:   9de54b7b7a89d9526b32305a331dea6a
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       vastuser: FULL_CONTROL
   x-amz-meta-s3cmd-attrs: atime:1561405102/ctime:1561405019/gid:1000/gname:vast                                               data/md5:9de54b7b7a89d9526b32305a331dea6a/mode:33204/mtime:1561405019/uid:1000/u                                               name:vastdata

Delete an Object

s3cmd [-c CONFIGFILE] del s3://BUCKET/OBJECT

or (alias for del)

s3cmd [-c CONFIGFILE] rm s3://BUCKET/OBJECT

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	Name of bucket
OBJECT	Name of object to delete

Example

Delete the file brown.txt from the bucket bucket1.

$ s3cmd -c vast.s3cfg rm s3://bucket1/brown.txt
delete: 's3://bucket1/brown.txt'

Set Object ACL

Before setting ACL permissions, we recommend you read Managing S3 Access Control Lists (ACLs).

s3cmd [-c CONFIGFILE] setacl s3://BUCKET/OBJECT [--acl-grant=PERMISSION:{USER_VAST_ID|GROUP}] [--acl-revoke=PERMISSION:{USER_VAST_ID|GROUP}]

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	Name of bucket
OBJECT	Name of object
--acl-grant	Grant stated permission for stated user or group.
--acl-revoke	Revoke stated permission for stated user or group
PERMISSION	Type of permission to grant: read write read_acp write_acp full_control all
USER	A user to which you want to grant the permission. Can be specified as: . A principal name in the format user@domain, where user is the user name and domain is configured for an external auth provider on the cluster (LDAP, NIS). Note Users on the VAST provider cannot be specified this way. A VID, which is a VAST ID used in the cluster's internal user database. A user VID is retrievable by running the user query VAST CLI command and specify udb as the context of the query. The output includes the user's VID.
GROUP	A group to which you want to grant the permission. The group can be specified as follows: Set the type to GroupLoginName and specify the group in the format group@domain, where group is the group name and domain is configured for an external auth provider on the cluster (such as LDAP). Specify the group's VID, which is the VAST ID used in the cluster's internal user database. A group VID can be retrieved by running the group query VAST CLI command and specify udb as the context of the query. The output includes the group's VID. If you want to grant permissions to an S3 predefined group, specify one of the following: `http://acs.amazonaws.com/groups/global/AllUsers`, or `http://acs.amazonaws.com/groups/global/AuthenticatedUsers`

Example

Grant write permission to object myfile.asp in bucket mybucket to the user whose VID is 6.

$ s3cmd -c vast.s3cfg setacl s3://mybucket/myfile.asp --acl-grant=write:6

Grant read permission to object myfile.asp in bucket mybucket to group mygroup@domain.com:

$ s3cmd -c vast.s3cfg setacl s3://mybucket/myfile.asp --acl-grant=read:'groupLoginName=mygroup@domain.com'

Return Object ACL

s3cmd [-c CONFIGFILE] info s3://BUCKET[/OBJECT]

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	Name of bucket in which object is stored
OBJECT	Name of object for which to return ACL

Example

Get ACL of file vast.s3cfg as part of full metadata output.

$ s3cmd -c vast.s3cfg info s3://bucket1/vast.s3cfg
s3://bucket1/vast.s3cfg (object):
   File size: 176
   Last mod:  Mon, 24 Jun 2019 19:40:27 GMT
   MIME type: text/plain
   Storage:   STANDARD
   MD5 sum:   9de54b7b7a89d9526b32305a331dea6a
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       vastuser: FULL_CONTROL
   x-amz-meta-s3cmd-attrs: atime:1561405102/ctime:1561405019/gid:1000/gname:vast                                               data/md5:9de54b7b7a89d9526b32305a331dea6a/mode:33204/mtime:1561405019/uid:1000/u                                               name:vastdata

Initiate a Multipart Upload

Multipart uploads are automatically used when a file to upload is larger than 15MB. The file is split into multiple parts, with each part of 15MB in size (the last part can be smaller). Each part is uploaded separately. The file is reconstructed at the destination when the transfer is completed. If an upload of a part fails, it can be restarted without affecting any of the other parts already uploaded.

s3cmd [-c CONFIGFILE] put [--multipart-chunk-size-mb=SIZE] FILE  s3://BUCKET[/PREFIX]

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
SIZE	Size of each chunk of a multipart upload. Files bigger than SIZE are automatically uploaded as multithreaded-multipart, smaller files are uploaded using the traditional method. SIZE is in Mega-Bytes, default chunk size is 15MB, minimum allowed chunk size is 5MB, maximum is 5GB.
FILE	A file over 15 MB in size. (If smaller, the file will be uploaded without being split into parts.)
BUCKET	The bucket to which to upload the file.
PREFIX	Prefix to append to the object name from which file system hierarchy can be inferred. When listing objects in the bucket, the prefix can be used to filter the list. You can specify a nested series of prefixes delimited by the delimiter (which is typically '/'). Object listing can be filtered by prefixes and delimiters. See List Objects in a Bucket. For example, when uploading a file called file.asp, if you specify a prefix /dev/misc the complete S3 object key becomes dev/misc/file.asp. A file system hierarchy can be inferred consisting of a folder called misc nested under a folder called dev in which the file file.asp is contained.

Example

Start uploading the large file Big.zip in parts of 500 MB:

$ s3cmd -c  vast.s3cfg put --multipart-chunk-size-mb=500 Big.zip s3://mybucket/
upload: 'Big.zip' > 's3://mybucket/Big.zip' [part 1 of 16, 500MB] [1 of 1]

Abort a Multipart Upload

After a multipart upload is aborted, no additional parts can be uploaded using the upload ID of the multipart upload. The storage consumed by any previously uploaded parts will be freed. To verify that all parts have been removed, see List Uploaded Parts for a Specific Multipart Upload.

s3cmd [-c CONFIGFILE] abortmp s3://BUCKET/OBJECT ID

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	Name of bucket to which object is being uploaded
OBJECT	Name of object which started uploading as multipart upload
ID	ID of the multipart upload

Example

If we kill the upload process mid-upload (ctrl-C), the upload ID is provided:

upload: 'Big.zip' -> 's3://mybucket/Big.zip'  [part 5 of 16, 500MB] [1 of 1]
  13041664 of 524288000     2% in  109s   116.48 kB/s^CERROR: 
Upload of 'Big.zip' part 5 failed. Use
  s3cmd abortmp s3://mybucket/Big.zip Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZJ_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM
to abort the upload, or
  s3cmd --upload-id Y7CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZK_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM put ...
to continue the upload.

Now we can use this upload ID to abort the upload:

$ s3cmd -c vast.s3cfg abortmp s3://mybucket/Big.zip Y8CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y78LZgx5H.8JJybYtMUUsW2GBXByAGiZK_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM

List Uploaded Parts for a Specific Multipart Upload

s3cmd [-c CONFIGFILE] listmp s3://BUCKET/OBJECT ID

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	The bucket the object was uploaded to
OBJECT	The name of the object being uploaded
ID	The ID of the multipart upload

Example

List uploaded parts for specific initiated upload of Big.zip to mybucket.

$ s3cmd -c vast.s3cfg listmp s3://mybucket/Big.zip Y7CYFHWCmnT6WUIw9nPTyU1AyseDrvsXhroXqVHIfA5AaTsUWw01Y8LZgx5H.8JJybYtMUUsW2GBXByAGiZK_lc3qtBDa2WO5x2F6397UHZSBwaj61P4pkw67zQF63nM

List Multipart Uploads

s3cmd [-c CONFIGFILE] multipart s3://BUCKET [ID]

CONFIGFILE	Configuration file. Defaults to $HOME/.s3cfg
BUCKET	The name of a bucket (multipart uploads to the specified bucket are listed)
ID	The ID of a specific multipart upload (for filtering list)

Example

To list all current multipart uploads to bucket mybucket in progress

$ s3cmd -c vast.s3cfg multipart s3://mybucket

Configure Bucket Logging

To enable bucket logging of access to bucket mybucket to a bucket named destination-bucket so that log object keys are prefixed with logs:

s3cmd accesslog --access-logging-target-prefix=s3://destination-bucket/logs/ s3://mybucket

Documentation Index

S3cmd

Using S3cmd with VAST Cluster S3

Install and Configure S3cmd

Minimal Configuration File Content

Additional Configuration Options

For HTTPS Connections

Signature Versions

Create a Bucket

Example

List All Buckets on the Server

Example

List Objects in a Bucket

Example

Retrieve Bucket Metadata

Example

Delete a Bucket

Example

Set Bucket ACL

Examples

Return Bucket ACL

Example

Upload an Object

Example

Create a Copy of an Object

Example

Download an Object

Example

Retrieve Object Metadata

Example

Delete an Object

Example

Set Object ACL

Example

Return Object ACL

Example

Initiate a Multipart Upload

Example

Abort a Multipart Upload

Example

List Uploaded Parts for a Specific Multipart Upload

Example

List Multipart Uploads

Example

Configure Bucket Logging