Rsync to AWS S3 bucket

Solution 1:

To communicate to s3 you need to have 2 things

1. IAM user credentials who has read-write access to s3 bucket.
2. A client like aws-cli for bash, boto library for python etc.

once you have both, you can transfer any file from your machine to s3 and from s3 to your machine. Below is the example for aws-cli.

1. to sync all files in a folder

   aws s3 sync source_folder s3://your_bucket_name/destination_folder/

2. to copy one file to s3

   aws s3 cp source_file s3://your_bucket_name/destination_folder/

just replace source & destination in the command to download any file from s3.

for more info follow aws docs

Solution 2:

You can try rclone.

Setup (docs):

rclone config

Sync (docs):

rclone sync /my/local/folder s3service:bucket-name

---

Solution 3:

aws-cli is great for syncing to S3 if you are not interested in permission.

Keep in mind that it does not keep permisión information nor other metadata that the files can have.

It is no well advice in the documentation, and thet you have surprises when restoring.

May be it is not important to sync your collection of music of photos, but it is of crucial importance if you are using it to backups your system.

There is another program s3cmd that keeps this data and can do a similar job. It is slower. But I have not tried it.

I was told that the problem is that you can keep in high costs as in S3 you do not only pay per MB saved, but for requests and rsync makes a lot of them.