Here's a video on deleting a bucket with a large number objects I just watched & found informative. It basically says something like you disable versioning, expire objects after 1 day, delete deletion markers, and delete deleted deletion markers (getting a little meta there).

https://youtu.be/TPavkqp-Ymo

You can nuke a whole S3 bucket. There's a switch in the cli and a checkbox in the GUI. I think there's another option for deletion protection that disables that somewhere else

I can't find the CLI option, the docs still say that the bucket must be empty[1]. There's something available in the management console but that doesn't really help with automated deployments and testing.

[1] - https://docs.aws.amazon.com/cli/latest/reference/s3api/delet...

Edit: Found aws s3 rb --force, but this just automates the process of listing/deleting objects then deleting the bucket. When you have 7 million objects in a cloudtrail bucket, that can take days.

No, you can nuke a bucket with stuff in it. I just realize I've never done this before, but the deletion screen calls out that all the contents will go away too.

https://i.imgur.com/j1av6iE.png

That screen will first delete all the items. It just issues the api calls for you. The amazon web interface is built on the API. It can’t do anything that you can’t do via the API.

Not all of those APIs are published or available via standard libraries...particularly with newer services.

That must be a somewhat recent addition - a year or two ago it wasn't possible (and the suggestion was what was posted in another comment - set the lifecycle rules to expire all the objects in a day)

edit: actually the "don't close your browser window" warning leads me to believe that's just a client-side JavaScript solution which doesn't work with massive buckets (we had one with millions of objects)

Why would the Javascript solution be inherently unable to deal with large buckets? Do we have to assume that they're taking some incredibly naive approach and trying to load every single ID into memory at once instead of streaming them out in chunks? It seems like Amazon should be capable of hiring an engineer who knows what they are doing.

It just seems like an inherently slow process. I remember at the time there was a python script someone wrote, which used the API to delete them in 1000 object chunks, and even running that with 10 threads the delete was going to take days and days.

The problem is the duration and durability of the exercise when you have millions of objects in the bucket. It could easily take days.

At least it's the kind of problem you can come back to and restart if the process dies in the middle. You're bulk deleting data so maintaining integrity isn't a concern.

This isn't strictly true - If you use the S3 dashboard the Delete Bucket button will abstract all the individual object deletes away for you.