I guess it was really so bad. Everything you have to build for yourself in the cloud is something you probably needed in your self-hosted stack -- a directory of users, permissions (and escalation), firewall rules, SSH key material, etc. I feel like no matter what production stack you choose, you're always in that "6 of one, half-dozen of another" state. Lots of stuff to interfere with getting your code out there and keeping it running.
(My real question is why the cloud providers don't offer these things for you -- they give you the tools to build solutions, not solutions. But I just want to work on my app. I wonder why things like App Engine never really caught on. I guess being opinionated drives more people away than it brings in with the promise of more simplicity, though to some extent App Engine was too simple, which is always bad. Meanwhile, what cloud providers offer certainly lets you bring your own opinion and build it out yourself. I wish there was a comfortable middle ground.)
I guess there’s probably not much demand for this kind of thing at large scales.
Hmmm, well the first generation of a cloud providers offerings has to be the bare necessities (EC2, VPC, IAM RDS etc). Then you can start providing a second gen of services that use the first gen, like AWS's ECS, and then a third gen like Fargate.
Feels like it is a very solvable problem, though. Only a matter of time.
I mean, fantastic, but what does this resource even represent, and what do I put in the string? It’s clearly an enum.
If it's of any help, I can promise you that creating a Terraform configuration to set this up will improve your life, happiness and general well being in the future.
Yes, Terraform has a learning curve, but it's also a black magic effery for the cloud.
Yes, Terraform/HCL can sometimes feel like the instructions for the holy hand grenade.
AWS can definitely be poorly documented and buggy at times.
I will never forget the story about the expert Amazon engineer that lost 80K USD.
if yoyr workload isnt simple though, then its time to pay the price until a better abstraction comes along ^.^
A great example is Cloudtrail. The best practice is to send your OrgTrail (an org-wide Cloudtrail) to a separate account for security reasons. Cool. Okay, sounds easy enough. The Splunk docs are useless for AWS, so consulting with YouTube, Reddit, etc is the go-to for this.
It's so much easier to just leave the OrgTrail logging into the management account it's not even funny.
My idea is to build simpler system on top of cloud networking and IAM/permissions.
Why would I want to have networking between components that are permissioned to interact with each other, and the other way, why some app/lambda's service account have permission to interact with particular database, but be blacklisted on firewall? Have GUI on top to drag and drop "routes" between apps.
Of course, make it optional etc, but I think it would save tons of devops time for smaller organisations without audit requirements unlike fintechs, medical stuff etc.
e.g. You’ve defined a nice, fully-functional end-state, which was generated n time ago, but now you need to make a change to that state, so you run your
terraform plan
That may be okay, but you should probably measure at least 3 times before you cut.
To scale up our reporting I thought it’s a good idea to offload report generation to a lambda function. Read from MySQL, dump into S3 and mail a link to that S3 file, make the link expire after couple of hours. Reasonably simple enough. But by God was it painful!! IAM is a rabbit hole, now later VPC restrictions on top and couple that with terribly organized document it makes no fun at all.
I get the need for all those layers of security but without well organized document everyone has to go through the ordeal. What we need now is plethora of how-tos for these common architectural patterns.