You're using S3 for the first time. Twilio has the capability to hire Ops staff that have used S3 for 10+ years. Throw a penny down the street in any major city and you'll hit a DevOps person with AWS experience. It's amazing to me how often I hear a company not set up s3 properly, or not enable budget alerts on AWS, etc. This is all day 1 AWS stuff. If that means taking away the ability for your developers to even create buckets then do so. A lot of you should just focus on development work and stay far away from operations/sec tasks, and that's fine. I'm going to assume most of these buckets aren't even made via code and are manual. That's bad enough.

I understand someone brand new to S3 being in the dark but someone in your PRs approval chain should have enough Ops skill to know to check these things if you're not going to hire an actual Ops person. And no, none of this is hard for anyone who has ever touched S3. Ops teams have known about these issues for over a decade. It's practically a meme at this point.

I understand what you're saying but you do realize that a huge number of software shops don't have massive teams or even dedicated OPs people right?

The number of heads on my project is ...4. We're it. There is no place or person to just push the work off to. There is no PR approval chain. We regularly meet to sync and discuss plans and thats it.

Until very recently all of our infrastructure was in house, right down the hall from me. I (or whoever was in that day) took care of making sure it was running smoothly.

Cloud is new to us, and there is no budget to hire a guy who manages AWS full time. That is the case at an insanely high percentage of software shops. Heck I would wager most people outside of 'new tech (sorry, I can't think of a better term)' shops are using point & click in the web UI to manage AWS. There is no code in source control that manages our infrastructure. I'm jealous of the people that get to do it all the 'right way' but for many there isn't enough manpower or time in the day to do it.

Now why we're in this position in the first place is another debate about why management thinks we need to go cloud at all. But its reality for many.

Oh trust me I know totally what you mean, I'm usually the first ops person hired at a startup once they've gotten the budget for one. My goal when I work with devs is to get as much of the infrastructure frustration out of their way as possible (unless they want to help with it) and automate the processes that slow down their code from getting deployed. This usually comes with months of fixing cruft/security issues, etc that could've lead to a "hack" like this when just the devs were running the ship.

I wish MORE small dev teams brought in ops people and didn't wait until they had a staff of 20-30 or whatever it typically winds up being. But I know that's a pipe dream and you're basically losing a developer salary to hire a devops/ops person which isn't the direction they want to go.

edit: typos

I totally understand where this is coming from, but even if we factor AWS out. IAM _IS_ difficult to implement, there really isn't a way to make it "simple" while still make it extremely granular. You can't have without the other.

To me this is like buying a high end sports car, and neglecting the maintenance because you never factored in the expensive maintenance, and therefore the true cost of what you're buying.

I agree. People do it all the time though.

The problem with S3 is the exact same problem with everything else in AWS. In their quest to be compatible and attractive to every single possible niche use case of the world's largest companies, they completely forgot the lone developer who just wants to upload a couple files.

You'll even see this in their sales pitches. AWS will spend so much time talking about the most advanced use cases and how they are possible, but if you ask them a simple question about how to run a single bucket that doesn't also involve using all of their ML tools and global accelerator and cloudfront, they'll be blindsided. It's almost as if they consider the common use cases to be "too simple/obvious" and therefor they never bothered to create any documentation or put any thought into it.

Your point is good, but one could suppose Twilio is not a "lone developer".

Certainly not, and in their layers of defense against this stuff, they should have had processes to train people, audit bucket policies, conduct penetration tests etc, and there's no excuse for them not having caught it before.

But with that said, one of those layers of defense is also using tools that are easy to keep secure so that the vulnerability doesn't appear in the first place. While Twilio isn't "a lone developer", I wouldn't be surprised if the person who did create that bucket on behalf of Twilio was just a lone developer fumbling around in the console. And again, that's no excuse, but it still is an area for improvement.

I kind of miss the times when working with the cloud was easier or more straightforward.

Now, as has been stated before, reading any cloud vendor documentation is like reading the Oracle manuals of years ago, not a happy experience.

Granted, these clouds now do things much more powerful and target more difficult enterprise scenarios. One can miss Heroku, and AWS has Elastic Beanstalk for that, but EB gets much more difficult fast.

It's gotten better, if you can believe it. Regardless, I've managed to come up with one foolproof method for securing my data: Never put anything important on S3.