AWS CLI

AWS CLI is a command-line interface for working with Amazon services. It provides a set of commands for managing files in cloud storage.

Installation

Copy link

To install AWS CLI on Ubuntu, use the command:

sudo apt install awscli -y

After installation, you can check the AWS version with the command:

aws --version

You can find information on installing AWS CLI on other operating systems in the AWS documentation.

Configuration

Copy link

To configure AWS to work with our S3 object storage, use the command:

aws configure

An interactive prompt will appear, asking for connection credentials. Enter them as follows, using the values from your bucket settings in your dashboard:

AWS Access Key ID [None]: <Access Key>
AWS Secret Access Key [None]: <Secret Access Key>
Default region name [None]: us-2
Default output format [None]: json

This will create a hidden .aws directory with config and credentials files, which contain AWS settings and authorization keys, respectively.

Working with Files

Copy link

Retrieving a List of Files in a Bucket

Copy link

To retrieve a list of files, use the ls command:

aws s3 ls s3://<bucket-name> --endpoint-url https://s3.hmstorage.net

Uploading a Local File to the Bucket

Copy link

To upload files, use the cp command. In the example below, the test.txt file is uploaded:

aws s3 cp test.txt s3://<bucket-name> --endpoint-url https://s3.hmstorage.net

Deleting a File in the Bucket

Copy link

To delete a file, use the rm command. In the example below, we delete test.txt:

aws s3 rm s3://<bucket-name>/test.txt --endpoint-url https://s3.hmstorage.net

Synchronizing Files

Copy link

To sync files between a local directory (in this example, the current directory .) and the bucket, use the sync command:

aws s3 sync . s3://<bucket-name> --endpoint-url https://s3.hmstorage.net

You can find detailed information on these and other commands in the AWS documentation.

Multipart Upload

Copy link

To perform a multipart upload using the AWS CLI, you need to use the aws s3api command. This process involves creating and completing a multipart upload in a few steps.

Preparation

Copy link

Before performing a multipart upload, the file must be split into parts. You can use the split utility to divide the file:

split -b <part_size> <path_to_file> <part_name_prefix>

• <part_size>: The size of each part (e.g., for 5 MB, use 5M).

• <path_to_file>: Path to the source file.

• <part_name_prefix>: Prefix for part names. For example, if you specify part, the parts will be named partaa, partab, and so on.

Step 1. Initialize the Upload

Copy link

Start the multipart upload to get an UploadId, which is needed for subsequent steps:

aws s3api create-multipart-upload --bucket <bucket_name> --endpoint-url https://s3.hmstorage.net --key <file_name_in_bucket>

The command's output will include an UploadId. Save it, as it will be used for uploading parts and completing the process.

Step 2. Upload Parts

Copy link

Use the upload-part command to upload each file part. You need to specify the UploadId, PartNumber, and the file part path. For example, to upload the first part:

aws s3api upload-part --bucket <bucket_name> --endpoint-url https://s3.hmstorage.net --key <file_name_in_bucket> --part-number 1 --body <path_to_part> --upload-id <UploadId>

Repeat this command for each part of the file, incrementing the --part-number and updating --body with the appropriate file part.

Step 3. Complete the Upload

Copy link

After all parts are uploaded, use the complete-multipart-upload command. First, create a JSON file with details about the uploaded parts, including the ETag and PartNumber:

{
  "Parts": [
    {
      "ETag": "<ETag_of_part_1>",
      "PartNumber": 1
    },
    {
      "ETag": "<ETag_of_part_2>",
      "PartNumber": 2
    }
  ]
}

Then, complete the upload:

aws s3api complete-multipart-upload --bucket <bucket_name> --endpoint-url https://s3.hmstorage.net --key <file_name_in_bucket> --upload-id <UploadId> --multipart-upload file://<path_to_JSON_file>

Abort the Upload

Copy link

If you want to cancel the upload, use the following command:

aws s3api abort-multipart-upload --bucket <bucket_name> --endpoint-url https://s3.hmstorage.net --key <file_name_in_bucket> --upload-id <UploadId>

List Multipart Upload Parts

Copy link

To check the parts uploaded so far, use the list-parts command:

aws s3api list-parts --bucket <bucket_name> --endpoint-url https://s3.hmstorage.net --key <file_name_in_bucket> --upload-id <UploadId>

Presigned URLs

Copy link

Presigned URLs allow you to create temporary links for accessing private objects in S3. These links provide access to an object without requiring authentication, and unlike changing an object's ACL, presigned URLs always have an expiration time (maximum 7 days).

Generating a Presigned URL

Copy link

Use the presign command to generate a temporary link:

aws s3 presign s3://<bucket_name>/<file_path> --endpoint-url https://s3.hmstorage.net

To set a custom expiration time, use the --expires-in flag, which specifies the time in seconds (default is 3600 seconds):

aws s3 presign s3://<bucket_name>/<file_path> --expires-in <time_in_seconds> --endpoint-url https://s3.hmstorage.net

How Presigned URLs Work

Copy link

Presigned URLs use AWS’s standard authorization mechanism. The difference is that the authorization values, typically sent as headers, are appended to the URL as query parameters. This allows external users to access the link via a browser or other client without additional setup. For example, a generated link might look like this:

https://s3.hmstorage.net/841b4a72-presigned/README.md?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=4AP1MJKIJFLUJN9ABQZ1%2F20241113%2Fus-2%2Fs3%2Faws4_request&X-Amz-Date=20241113T104406Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=737a677e78f88ab989b1c9b8edb454f70ea8da42e6b79774e68aa0fc6a52cb81