Before using AWS I was in the 'what morons!' camp whenever some company got breached via leaving things wide open on AWS. Now I'm in the 'ok I understand why its so common' camp.
I understand someone brand new to S3 being in the dark but someone in your PRs approval chain should have enough Ops skill to know to check these things if you're not going to hire an actual Ops person. And no, none of this is hard for anyone who has ever touched S3. Ops teams have known about these issues for over a decade. It's practically a meme at this point.
The number of heads on my project is ...4. We're it. There is no place or person to just push the work off to. There is no PR approval chain. We regularly meet to sync and discuss plans and thats it.
Until very recently all of our infrastructure was in house, right down the hall from me. I (or whoever was in that day) took care of making sure it was running smoothly.
Cloud is new to us, and there is no budget to hire a guy who manages AWS full time. That is the case at an insanely high percentage of software shops. Heck I would wager most people outside of 'new tech (sorry, I can't think of a better term)' shops are using point & click in the web UI to manage AWS. There is no code in source control that manages our infrastructure. I'm jealous of the people that get to do it all the 'right way' but for many there isn't enough manpower or time in the day to do it.
Now why we're in this position in the first place is another debate about why management thinks we need to go cloud at all. But its reality for many.
I wish MORE small dev teams brought in ops people and didn't wait until they had a staff of 20-30 or whatever it typically winds up being. But I know that's a pipe dream and you're basically losing a developer salary to hire a devops/ops person which isn't the direction they want to go.
edit: typos
You'll even see this in their sales pitches. AWS will spend so much time talking about the most advanced use cases and how they are possible, but if you ask them a simple question about how to run a single bucket that doesn't also involve using all of their ML tools and global accelerator and cloudfront, they'll be blindsided. It's almost as if they consider the common use cases to be "too simple/obvious" and therefor they never bothered to create any documentation or put any thought into it.
But with that said, one of those layers of defense is also using tools that are easy to keep secure so that the vulnerability doesn't appear in the first place. While Twilio isn't "a lone developer", I wouldn't be surprised if the person who did create that bucket on behalf of Twilio was just a lone developer fumbling around in the console. And again, that's no excuse, but it still is an area for improvement.
Now, as has been stated before, reading any cloud vendor documentation is like reading the Oracle manuals of years ago, not a happy experience.
Granted, these clouds now do things much more powerful and target more difficult enterprise scenarios. One can miss Heroku, and AWS has Elastic Beanstalk for that, but EB gets much more difficult fast.
I'm sure an expert finds it all very natural. Security is always hard and has to be done right the first time. That's a domain that's always going to deter novices, and perhaps the only way forward is to hire an expert to ensure you're doing the right thing.
Still... I've been programming for four decades and it felt a little odd to be quite so out of my depth. It's not the same as anything I've done before, but it felt like it shouldn't have been quite so bewildering.
But, as others have said, the documentation at times is poor at best, and it can be easy to make mistakes.
Some preventives every ops team can take are:
1. Never make manual permission changes in the console. Everything is done via SCM’d code.
2. All permission changes are peer reviewed.
3. Create logical separations of buckets so it’s more obvious what the permissions should be of a bucket. You can go so far as to have separate accounts that are for public uploads, separate accounts for public assets, etc. Avoid granting elevated permissions at a folder level, which is harder to understand as a developer without access to view the ACLs.
Good luck everyone!
The first time I heard “infrastructure as code” I thought it was some dumb new buzz words, but no, it’s good advice. Your cloud formation, serverless, terraform whatever should be stored in GIT a stack replace is how changes should be made.
I consider the console as read-only.
The web console should be just another UI for editing the configuration in source control. As an example of why that's important, a UI can give you a warning and use affordances to guide beginners away from dangerous setups. In the case of S3, a slider for basic access level can use colors and show a list of objects that would become world readable or writable at a given level (for that matter why is world writable even an option?). A giant YAML file looks the same whether it's safe or dangerous.
If you really don't want to use a UI, then maybe abstracting a linting layer that feeds the exact same warnings to the UI as to your CI system and text editor would help.
Should. But isn’t.
So until it is, IaC for production. Console or CLI for development.
I do the same, and sometimes I need to do something manually so that I can grok what configurations I need to define in terraform / whatever other tool I'm using. POC'ing is different than a production or even staging service.
It would be much harder make that mistake with Google Cloud Storage, at least in the web console. GCS also supports disabling bucket ACLs permanently at bucket creation time, and that is the option they recommend[1].
AWS does scan and alert for some public bucket issues. But with all the public S3 issues, I think it is clear there needs to be more prominent warnings in the AWS console, the command line, and with tools like Terraform.
S3 does too. There's an entire page during the creation wizard that is dedicated to blocking any and all public access, even causing the bucket to ignore ACLs or other settings that would otherwise expose the bucket publicly. All public access is disabled by default, and enabling it actually requires the user to actively uncheck 5 different checkboxes, each of which explains that unchecking it will open the bucket up to public access, and then even requires an additional attestation, in a large orange warning box, that says you acknowledge that the options you chose will result in the bucket being public.
I'll be the first person to tell you that AWS is overcomplicated and hard to use, but this isn't that. IAM and bucket policies are a pain to work with, but if you screw up with those, the worse you will do is expose your bucket internally. But exposing a bucket publicly to the internet is an entirely different act, and there's not really any excuse for it other than just not reading the directions.
However, neither this option in S3 nor the option you linked in GCS would have served Twilio's use case. They wanted their bucket to be publicly accessible, just not publicly writable. That's an entirely different access management issue.
You already have a role in AWS that you use. Go into the console and say “this role can do S3 to this bucket” and you’re done.
If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL.
Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a single role, and you don’t have to worry about rogue roles exposing the bucket.
Perhaps some of the issue is that permission can be managed on both IAM and a bucket policy?