S3cmd

S3cmd is an S3 API client with a command-line interface, used to interact with S3-compatible storage.

Installation

To install S3cmd on Ubuntu, use the following command:

sudo apt install s3cmd -y

Configuration

To configure S3cmd to work with our S3 object storage, you need to create a .s3cfg file in your home directory with the following contents:

[default]
access_key = <Access Key>
secret_key = <Secret Access Key>
bucket_location = us-2
host_base = s3.hostman.com
host_bucket = s3.hostman.com
use_https = True

You can find Access Key and Secret Access Key in the bucket Settings under S3 Parameters.

Working with Files

Listing Files in a Bucket

To list the files in a bucket, use the ls command:

s3cmd ls s3://<bucket_name>

Uploading a Local File to a Bucket

To upload files, use the put command. In the example below, the file text.txt is uploaded:

s3cmd put test.txt s3://<bucket_name>

Multipart Upload

The S3cmd client supports multipart upload, which is automatically enabled for files larger than 15 MB. This method splits the file into multiple parts (chunks) and uploads them in parallel, speeding up large file transfers and improving fault tolerance.

If needed, multipart upload can be disabled using the --disable-multipart option.

To change the size of individual chunks, use the --multipart-chunk-size-mb=SIZE option, where SIZE is the size of each chunk in megabytes. By default, the chunk size is 15 MB, with a minimum allowable size of 5 MB and a maximum of 5 GB.

For example, to set the chunk size to 10 MB, use the following command:

s3cmd put <file_name> --multipart-chunk-size-mb=10 s3://<bucket_name>

This setting allows you to optimize large file uploads based on network bandwidth and performance requirements.

Managing Interrupted Multipart Uploads

If a multipart upload is interrupted, file fragments do not appear in the bucket but still occupy disk space on the server. To view a list of interrupted uploads and see which fragments are using space, use the following command:

s3cmd multipart s3://<bucket_name>

This command shows the date and time the upload started, the file path, and a unique upload identifier (ID).

To delete fragments from failed uploads, use this command:

s3cmd abortmp s3://<bucket_name>/<file_name> <upload_ID>

Downloading a File from a Bucket

To download a file from a bucket to your local device, use get:

s3cmd get s3://<bucket_name>/test.txt

Deleting a File in a Bucket

To delete a file, use the del command. In the example below, we delete the file text.txt:

s3cmd del s3://<bucket_name>/test.txt

File Synchronization

To synchronize files between a local directory (in this example, the current directory: .) and a bucket, use sync:

s3cmd sync . s3://<bucket_name>

For a full list of commands and additional information, consult the S3cmd documentation.