News for hackers. Stuff that matters.
Re:Backblaze Hard Drive Stats Q2 2019 by kalleboo on Tuesday August 06, 2019 @09:46AM (#20625097)
That must be a somewhat recent addition - a year or two ago it wasn't possible (and the suggestion was what was posted in another comment - set the lifecycle rules to expire all the objects in a day)

edit: actually the "don't close your browser window" warning leads me to believe that's just a client-side JavaScript solution which doesn't work with massive buckets (we had one with millions of objects)

Re:Backblaze Hard Drive Stats Q2 2019 by jandrese on Tuesday August 06, 2019 @11:58AM (#20626346)
Why would the Javascript solution be inherently unable to deal with large buckets? Do we have to assume that they're taking some incredibly naive approach and trying to load every single ID into memory at once instead of streaming them out in chunks? It seems like Amazon should be capable of hiring an engineer who knows what they are doing.
Re:Backblaze Hard Drive Stats Q2 2019 by kalleboo on Tuesday August 06, 2019 @12:10PM (#20626499)
It just seems like an inherently slow process. I remember at the time there was a python script someone wrote, which used the API to delete them in 1000 object chunks, and even running that with 10 threads the delete was going to take days and days.
Re:Backblaze Hard Drive Stats Q2 2019 by jcims on Tuesday August 06, 2019 @12:15PM (#20626546)
The problem is the duration and durability of the exercise when you have millions of objects in the bucket. It could easily take days.
Re:Backblaze Hard Drive Stats Q2 2019 by jandrese on Tuesday August 06, 2019 @01:26PM (#20627384)
At least it's the kind of problem you can come back to and restart if the process dies in the middle. You're bulk deleting data so maintaining integrity isn't a concern.