Data Transfer to S3 Storage

You can transfer data to our storage either automatically using the control panel's functionality or with third-party tools like Rclone. This article will cover both methods.

Automatic Data Transfer

You can transfer data from any S3 compatible storage to Hostman storage through your control panel. It also allows you to transfer data between buckets within our storage.

To initiate the transfer:

1. Go to the S3 Storage section, click on the bucket and open the Data Transfer tab.

2. Enter the connection credentials for the third-party storage. You can request this information from your current storage provider.

3. Click Start Transfer. Depending on the number of objects in the source bucket, the transfer time may vary. The progress of the copying will be displayed on the bucket page with a yellow indicator.

For migrating constantly changing data, we recommend combining this method with the manual synchronization described below.

Please note that the Data transfer option copies data and overwrites existing files. To avoid overwriting, you need to use synchronization.

Data Migration Using Rclone

This method is preferable for transferring large volumes of data (with millions of objects) and constantly changing data.

You will need:

• A bucker created in the Hostman S3 storage.
• A computer or virtual server with Rclone installed.

The process consists of several stages, which we will review in order.

Step 1. Preparation

Obtain the credentials for the source bucket from the third-party provider and configure Rclone. 

Add two profiles (remotes) to it—one with the credentials from the third-party provider and the other with the credentials for our storage. 

For examples in this article, we will use the profile names other-provider and hostman.

Step 2. Initial Data Copy

For large volumes of data, the process can take a significant amount of time, so it's advisable to run the copy command in a tmux or screen session. This allows you to keep the command running in the background and disconnect from the server.

Start a new tmux session with the command:

tmux new -s storage-migration

Start copying objects with the command below, entering your details:

rclone copy -v -P --log-file=rclone.log --transfers=128 [other-provider]:[bucket-name] hostman:[bucket-name]

The --transfers option specifies the number of parallel uploads. The more uploads, the faster the copying will occur; however, the maximum number of uploads is limited by the capabilities of the computer running the command. If errors occur, reduce the number.

You can disconnect from the tmux session by pressing the key combination Ctrl+B, then releasing the keys and pressing D. You will see the message [detached (from session storage-migration)].

You can reconnect to the active tmux session using the command:

tmux a

Step 3. Data Synchronization

Wait for the copying to complete and then start the synchronization:

rclone sync -v -P --log-file=rclone.log --transfers=128 [other-provider]:[bucket-name] hostman:[bucket-name]

Only changed or new files will be copied to the target bucket.

Step 4. Stopping the Service and Final Synchronization

To maintain data consistency, you will need to temporarily stop the service. During this time, run the final synchronization:

rclone sync -v -P --log-file=rclone.log --transfers=128 [other-provider]:[bucket-name] hostman:[bucket-name]

This synchronization should be faster than the previous one if not much time has passed and a large amount of new data has not accumulated.

Step 5. Configure the Service to Work with the New Storage

After the final synchronization of the storages, connect the services to the new storage. Once the services successfully switch to the new storage, you can delete the data from the old storage.