News for hackers. Stuff that matters.
Re:Moving 25TB data from one S3 bucket to another took 7 eng... by awiesenhofer on Sunday September 13, 2020 @06:28PM (#24464466)
I had a similar task once, tried it with awscli and s3cmd first, then took about 5 minutes to google for a multithreaded alternative and found s4cmd[1], problem solved. (tmux to keep the session running)

Makes me kinda doubt at least parts of this story - not one of these 7 engineers thought about googling?

Edit to add: Agree with the basic premise though - s3 can be real slow when when you actually need to work with your data.

[1] https://github.com/bloomreach/s4cmd

Re:Moving 25TB data from one S3 bucket to another took 7 eng... by gfody on Sunday September 13, 2020 @06:44PM (#24464560)
Re:Moving 25TB data from one S3 bucket to another took 7 eng... by miniman1337 on Sunday September 13, 2020 @08:18PM (#24465199)
s5cmd is super fast