Re:Backblaze Hard Drive Stats Q2 2019
by jandrese on Tuesday August 06, 2019 @11:58AM
(#20626346)
Why would the Javascript solution be inherently unable to deal with large buckets? Do we have to assume that they're taking some incredibly naive approach and trying to load every single ID into memory at once instead of streaming them out in chunks? It seems like Amazon should be capable of hiring an engineer who knows what they are doing.
[ Parent
| on HN ]
It just seems like an inherently slow process. I remember at the time there was a python script someone wrote, which used the API to delete them in 1000 object chunks, and even running that with 10 threads the delete was going to take days and days.
The problem is the duration and durability of the exercise when you have millions of objects in the bucket. It could easily take days.