A decade ago or so AWS was marketed as - to simplify - "fire all your sysadmins because we manage the systems for you" to the current state where good AWS DevOps engineers are expensive and not so easy to find and you pay through the nose if you really follow AWS recommendations, especially for larger setups.
CTOs bought the "fire all your sysadmins" literally, and now we have a 20% overhead on all app devs who are supposed to learn all this stuff.
It's crazy to me. It's literally the opposite of every management training / book about focussing on highest value work / where you add value / where you provide expertise.
Unless you're using AWS as yet another VPS provider, creating a "cloudy" product requires way too much knowledge about AWS services as a developer.
I have clients, product, scrum masters, tech leads, and others breathing down my neck on feature delivery, and you're asking me to figure this alphabet soup out, lol.. don't expect happy results dudes.
I've been a domain expert in a field for 20 years, what value do I bring to the client in dealing with this alphabet soup of AWS networking/security? Either it works and client gets what they want, or it doesn't and they don't.
This stuff is a full time job and should be respected as such. If you think you can push it all down on app dev, you are going to have unhappy results.
https://www.lastweekinaws.com/blog/an-alternat-future-we-now...
I wouldn't use it in production (some companies do, for internal things) but the very fact it exists speaks volumes.
However, I think that customers should have the option once they sign some stuff acknowledging that they may be really, really screwed similar to Bad Things done as a root user in AWS.
Given the distributed nature of so many requests in AWS I’d imagine that a GraphQL-like setup would be conceptually compatible with such a system. It’s not terribly far from tracing systems like OTEL
System managed identities + AAD is an absolute joy in my experience on Azure. I can grant a function app access to a SQL database by simply using the literal name of the function app (qualified by the object's identifier if aliased). No passwords, certificates, etc. involved anymore. It literally just works.
Combine their auth package with things like application insights, and you are not really ever stuck thinking "WTF happened", assuming you have enough awareness to look at those logs.
> I guess by keeping such a shitshow they sell more support contracts though.
This is also a problem with Azure. You really have to know where all of the mines are located before you start walking through their field or you will wind up in a similar state of contorted frustration. I'd argue that Microsoft leaves substantially better hints regarding where mines might be, but they still conveniently leave out enough hints such that only a wizard with 2 decades of experience can pierce the final veil of bullshit and avoid a 5-10x multiple on cost & complexity.
Also, what's up with the way almost every any change in Azure takes some random period, sometimes hours, to take effect and there's not necessarily any indication that anything is happening.
I can't comment on the general correctness of your comparison (never having used Azure), but this particular one is flawed, since the same is true in CDK. That said, there's enough arcane weirdness in AWS' configuration requirements that I suspect your point is _broadly_ correct.
In only a few toy scenarios!
For example:
Azure SQL Database, Managed Instance, and SQL 2022 in a VM all support AAD authentication.
Okay.
SQL 2022 supports only user accounts and not managed identities.
SQL Managed Instance requires an additional global AAD permission button to be pressed post-deployment or it won’t work. This can’t be templated.
The first two have AAD permissions controlled via their template (ARM API) — but only exactly one full admin group. All other permissions must be assigned via T-SQL. Again, can’t be templated via ARM. (Microsoft has better data plane RBAC for random open source products than their own flagship database!)
User access audit logs are easily collected for Azure SQL Database, needs manual T-SQL for SQL MI and is missing in action for SQL Server.
This is just SQL, which now at least supports user-assigned managed identities… if you include the latest SQL client which has a breaking namespace change and hence won’t work with third-party code. It also drags in 500 MB of dependencies.
Lightweight functions, you were saying? Hmmm…
Anyway, I got fed up with the whole thing when I tried hooking up a Storage Account in the same way and discovered that it didn’t support UMSI in connection strings — you need two code paths for dev-time and runtime!
Up until recently the whole thing was an unstable mess. Authentication would take 5 seconds in some scenarios and then wouldn’t cache the tickets properly.
It turns out that every Azure team is implementing the whole thing separately! It’s not at all like traditional AD where the platform provides the authentication functions in a uniform way and apps can just pick up the ambient tokens.
No, in this shiny new world every service does RBAC differently, connection strings differently, auditing differently, caching, config, and on and on.
All of this within 100% Microsoft products and services.
It’s a poorly thought out mess.
I am not sure I understand. The "toys" you describe are all we need to get the job done. Azure SQL Hyperscale, Azure Functions C#/v4, AAD, another region for DR (SQL geo replica + Function App x2) and that's about it... Why do we want to make this hard?
If you want to try to force the square peg through the round hole, I get it. It's going to be a clusterfuck as you describe. But, I decided we were going to get completely trashed on the Microsoft kool-aid so we don't have to worry about 3rd party clients and other non-happy-path integrations. I click 1 big dumb button in my Visual Studio client and some time later my app is running on a live URL completely managed by Microsoft. This is a pretty good life. I don't have patience for things like I used to. Leadership & investors are not giving me much to work with these days.
I work in a job where I don't get to make product decisions, I just have to "make it work" on the cloud.
Despite this, the only applications I've worked on in the last three years have all been ASP.NET web apps with SQL databases. Even with this pretty vanilla combo, I found hundreds of bugs or interoperability issues in Azure. These aren't esoteric things, they're not round pegs and square holes. These are Microsoft-shaped pegs that ought to fit into Microsoft-shaped holes.
Things get messy the instant you step off the "tech demo MVP" happy path.
Like I said: Third-party code that talks to SQL, Storage Accounts, or SQL databases other than the one of their two Platform-as-a-Service offerings!
Even if you stick to just functions and Azure SQL Databases, there are still limitations and issues.
Templating is painful or impossible. Data-plane RBAC is missing in action. Etc...
If you're a solo developer working a tiny app deployed with click-ops to a single region, then I suppose it is "fine".
For everyone else, it's not so fine...
not made easier by the fact that debugging service accounts can sometimes require rebooting boxes or clusters
'cloud traceroute' to discover where exactly the problem is would be amazing
(in fairness there are 'least privilege' tools I haven't tried -- listed below and I recall reading about others)
1. iam access analyzer https://aws.amazon.com/blogs/security/iam-access-analyzer-ma...
2. airiam https://github.com/bridgecrewio/AirIAM
3. policy simulator https://cloud.google.com/policy-intelligence/docs/iam-simula...
I've found Athena doesn't appear to search through all log entries (possible user error on my part); I've been downloading the cloudtrail logs directly from S3 and grepping through the logs directly to find the relevant entries to figure out the permissions needed.
If you need to decode an encrypted error message, use `aws sts decode-authorization-message --encoded-message $MESSAGE`. It'll return a JSON string. I typically use `aws --profile limbledevadmin sts decode-authorization-message --output text --encoded-message $MSG | jq .context.action` to extract the needed permission.
I've wasted hours trying to work out why I get an unauthorised error, and the official docs say to manually pick your way through the 8 or so different policies that might apply (SCPs, IAM, resource policies, etc). Yeah right.
When you eventually find the page showing how to use Athena with Cloudtrail, you discover half the requests are inexplicably missing. And even though error messages include a request ID, you can't easily query for them, if it's even possible. Maybe it is and I just haven't discovered the right athena incantation. They should make it easy to just query by request ID and get clear messages telling you what policy denied the request.
It's a total train wreck from start to finish. I guess by keeping such a shitshow they sell more support contracts though.