News for hackers. Stuff that matters.

A comment on Moving 25TB data from one S3 bucket to another took 7 engineers and 2 full days. Parent comment

Re:Moving 25TB data from one S3 bucket to another took 7 eng... by fctorial on Monday September 14, 2020 @01:09AM (#24466962)
The source bucket had a ton of small files and most of the time was spent making network requests (O(<number of files>)). The suggested solution was to do a batch operation (open a ticket for a batch operation). Would rclone have helped in this scenario?
[ Parent | on HN ]
Re:Moving 25TB data from one S3 bucket to another took 7 eng... by darthShadow on Monday September 14, 2020 @01:48AM (#24467185)
Yep, as mentioned in the other comment, with configurable `--transfers` & `--checkers`, the only bottleneck is the amount of CPU you have available and can allocate.
[ Parent | on HN ]
Re:Moving 25TB data from one S3 bucket to another took 7 eng... by fctorial on Tuesday September 15, 2020 @11:17AM (#24482152)
> the only bottleneck is the amount of CPU you have available and can allocate.

In this case S3 will be the bottleneck. 25 TB will have around 700M files (assuming it's as dense a linux installation). S3 can do 5000 operations per second. It'll take at least 40 hrs (assuming everything works at peak speed during the whole process).

[ Parent | on HN ]
Re:Moving 25TB data from one S3 bucket to another took 7 eng... by oarsinsync on Monday September 14, 2020 @01:39AM (#24467130)
rclone lets you specify how many files to handle in parallel. It defaults to 4, and auto retries in the event of failures.

Ramp up the parallelism, and don’t make 7 people monitor it round the clock.

[ Parent | on HN ]