Re:Moving 25TB data from one S3 bucket to another took 7 eng...
by fctorial on Tuesday September 15, 2020 @11:17AM
(#24482152)
> the only bottleneck is the amount of CPU you have available and can allocate.
In this case S3 will be the bottleneck. 25 TB will have around 700M files (assuming it's as dense a linux installation). S3 can do 5000 operations per second. It'll take at least 40 hrs (assuming everything works at peak speed during the whole process).
In this case S3 will be the bottleneck. 25 TB will have around 700M files (assuming it's as dense a linux installation). S3 can do 5000 operations per second. It'll take at least 40 hrs (assuming everything works at peak speed during the whole process).