Some internal perspective - IAM has maybe thousands of options but fundamentally it is "what does this role have access to doing (action + resource)" + "who has access to this role". That is really it from a 10k foot level.

IAM is great because it applies internally just like it does externally. The internal AWS team don't get more access than you do, and if we get access to do certain thing on your account to perform specific service that's because you have a service principle in your IAM trust relationship that allowed us access, that you can see, and audit. For instance, lambdas have lambda role because you don't want lambda service just reading your S3 buckets because "we're AWS we automatically get access", you can absolutely see and control access, even if it is internal to AWS.

> but fundamentally it is "what does this role have access to doing (action + resource)" + "who has access to this role". That is really it from a 10k foot level.

Hahahaha. No, fundamentally it is one input into a huge mess that you cannot actually see or audit from a 10k foot level.

AWS has produced a long, rambling and imprecise description of (some of?) what’s actually going on. You can read it here:

https://docs.aws.amazon.com/IAM/latest/UserGuide/access_poli...

Some of what they’re describing doesn’t even live within the IAM umbrella as far as I can tell. I’m not convinced that a concise, formal and unambiguous specification exists anywhere, even within AWSes own development teams.

I’ve asked LLMs to write AWS “policy”. They get the grammar mostly right. They cannot explain what the effects are in a manner that they will stand by after they search the web for documentation. Since I have never found good documentation despite looking, I can’t personally do any better than the LLMs. I’d love to be pointed at real documentation or specs.

They are just some slight variation of the fundamental idea. For example resource policy and org SCP are just the same check on a different level (e.g. more of who has access to what). They are attached to Organization and individual resource respectively (vs Account) so they need to exist in a separate place. And then in use they are ALL checked before an access is granted.

I don't work for IAM but I worked for several other teams over the years and IAM is actually one of the least confusing services. But I am definitely biased and have more than average amount of experience on this particular subject. I still think the general idea is more sane than Azure Account for example. I do think this reflect on the philosophical level where whether cloud are building blocks or are they consulting projects. I personally think IAM is done right in that regard.

> And then in use they are ALL checked before an access is granted.

I know they’re all checked. What don’t know is how the results of those checks are combined to get the final result. As far as I can tell, the result is not something like OR or AND — it seems like it’s something exceedingly complex and that the output of the policy part may be more complex than just a Boolean value.

Maybe the underlying implementation is fantastic (and my distinct impression is that AWS takes this stuff far more seriously than Azure), but that doesn’t mean that the docs are easy to find or that the system actually makes sense in anything other than an agglomeration-of-backwards-compatible-layers sense.

The output is Allow or Deny. To be Allowed, you need an policy that includes an Allow statement. If you don't have one, then the result is implicit Deny by default. Also, any explicit Deny wins over all Allows. Disclosure: I work on AWS IAM.

> that's because you have a service principle in your IAM trust relationship that allowed us access

That’s why it’s so complicated!!!

I don’t understand how I should evaluate trust for your internal EBS org versus your internal ALB org.

I kinda just expect it to be all “AWS” trust.

And it’s all garbage anyway. There’s no way I can prevent the hypothetically untrustworthy EBS team from surreptitiously adding charges to my account if they want to. Right? This would maybe make some sense if I could top level turn off/on services, but that isn’t how it works.

—

I have no doubt this makes some sense from someone inside the machine, but from the outside it’s not helpful nor useful.

3 things to untangle here.

1. It's about trust and auditability, while you may not want or need it, there are a lot of customer that are either interested or legally obligated to know who have accessed certain data.

2. It's about dogfooding - how would you trust an identity and access system when the company does not even use it internally?

3. In general, there are quick buttons and template to do it if you don't want to worry about it, in the LLM age, this gets easier. Personally I prefer this because I intensely dislike "magic". This allow you to control, to the maximum degree possible, what is actually going on, despite not owning any of the physical aspect of the data center.

1. It's about imposing worst-case complexity on the 99% of people who will never benefit. 2. Some of that complexity only arises because of the dogfooding 3. No it doesn't get easier, because you still need to understand what those things actually do to know if they're right for your use case, and besides if you're driving everything from terraform then having a "quick button" is precisely useless.

We had an AWS rep try to sell us on an AI tool to help with predicting the IAM permissions that our infrastructure code needs. My response was, essentially, "why have you built a deterministic system so complicated that it needs an AI to configure correctly?" I have not had an answer.

That's all fine and good, but I still don't know how much trust the EBS team versus the ALB team.

And I don't think you do either.

You don't need to trust EBS or ALB team. You just have a contract that says whether ELB/ALB get access to your data/resources or not.

Also the access are not "team", it's software. The control plane would have access to resource if you granted the service principle and then it would go ahead to do what was already written as service logic for those resource. The engineers do not gain access to your resource now - all prod service principles are access gated and we can only see certain non-PII logs as required by operational events. We don't run the service manually.

>I kinda just expect it to be all “AWS” trust.

This would be very unwise from security standpoint. Internal access to customer stuff is granular and made hard for internal staff to gain, to minimize chances of screw up intentional or not.

I agree. Adding a service principal always raises an eyebrow for me, just a blanket "hey we're aws trust me bro" is a little bonkers.

How does this work in Azure and Google Cloud?

One thing i would like to see in IAM would be sonething like verb actions, currently, if you want to give least privilage, you have to trial and error your api call until you get it right. Since aws have a very good api definition on all consumers (rest, aws-cli, boto uses same strucyure), i think it would be doable.

I mean something like actions: s3:cp Resource: bucketarn/key

Most of the time, actions are self explanatory and good enough, but i recently gave a developer permission to scale an asg, and it required a lot of unguessable actions, if i were to give "actions: scale" (forgot the correct cli parameter for it), it would make more clean env

> fought against X adoption because it was "too complex", only to slowly reinvent X badly

This is a surprisingly common pattern in technology and software. Some things are definitively the “standard” at this point yet so many people simply refuse to spend the time to properly learn them.

> This is a surprisingly common pattern in technology and software. Some things are definitively the “standard”

It is also a surprisingly common pattern to adopt very complicated solutions for applications that are never going to need them

ultimately it is not possible to come up with a "standard" that is an acceptable replacement for good judgement

Another point is that while it can be more expensive than self hosting, the savings are dwarfed by the engineering costs. A decent infrastructure engineer working for 2 man months on your “money saving” ovh setup costs you more than you can possibly save by not just using fargate or rds whatever.

But unless you're on a PaaS, you have "infrastructure engineers" already. So why not at least let them make back their salary by making them built a cost-efficient infrastructure?

This is rarely actually true but it's a common falsehood told by people who have financial interest of keeping everyone on AWS.

And that includes engkneers that only know how to use AWS and are terrified at having to learn something else.

How much would you pay for 2 months of infrastructure engineer time? And how many millions / tens of millions / hundreds of millions are you imagining being spent on overpriced AWS services?

(Also, those AWS services are not engineering-free. I tried to migrate a system to RDS once and gave up after quite a few hours when I got to the part of the documentation that suggested that I edit my sql dump using sed to get it into a form that RDS would accept. No, thanks.)

IAM is unnecessarily bad. I recently had to set a trivial policy, and was doing it correctly.

The console kept warning me that I was giving root AWS access to my external application because they want people to use the locked in AWS path, and I was running off cloud.

On top of that, they break copy paste on the web console, so you can’t just ctrl-c ctrl-v and then ask Claude to explain their WTF-ery. Instead, you have to OCR or send a PNG.

I honestly did not think they could make IAM worse, yet here we are. Bastards.

I guess I should also point out that I’ve used AWS at extremely large scale in the past, which is why I’m running this subproject on another cloud.

As for simple permissions, go read the UNIX paper. It spends a page or two on their approach and is all you need.

Then, read the paper on mapping between NTFS SMB ACLs and NFS. It’s either impossible or undecidable, depending on the deployment. IAM is from the windows acl lineage which is known pessimal from a usability and security perspective.

IAM is NOT from any lineage. It has grown organically and is complicated, just as any other policy language. AWS even uses an automatic proof assistant to verify IAM policies.

However, the secret to IAM in AWS is to NOT use IAM. Just create separate AWS accounts for separate services and only share whatever resources are needed. Then you can have dead simple IAM policies because you won't need to do granular permissions ("AWS role X can access database Y").

> Just create separate AWS accounts for separate services

My understanding is that different AWS accounts have different mappings of availability zones, so it's very easy to suddenly find yourself with an unexpected bandwidth bill due to all the cross-az traffic.

I've been irritated at AWS (and the other large cloud providers) that they charge $0.01/GB for cross-az traffic. That's $3.24/Mbps -- about the same I was paying for internet transit (as in: from London to anywhere in the world) 20 years ago, and this is just between two datacenters in the same city controlled by the same organisation, markup must be 10,000x or more considering these places are cross-connected with massive bundles of fiber!

> My understanding is that different AWS accounts have different mappings of availability zones, so it's very easy to suddenly find yourself with an unexpected bandwidth bill due to all the cross-az traffic.

As far as I remember, accounts within the same organization will have the same mapping. You also can use stable zone names these days, instead of the regular mappings.

And yeah, egress traffic pricing is freaking insane at this point. It's the biggest reason to NOT use AWS.

Insanely high S3 storage charges too. $23/TB/month? Even with the insane HDD pricing that we see today, that's paying off a drive in 1 month (at retail) that will last for 50-100 months. Sure, there's probably some encoding overhead, but it's still mad.

S3 is pretty competitive if you want similarly-performing storage with consistent millisecond-level latency, high scalability, and at least 3x redundancy. Try looking at how much it's going to cost you in enterprise SSDs :)

Is it 3x redundancy forever? I always just kinda assumed it was RS encoded after a while, so only 30-50% larger than a single copy. Plus, almost all object storage is written to / read from hard disks, not to SSDs. Unless they're in a caching layer that is.

I know Azure has done a bunch of work around Pyramid Codes (essentially a locally repairable EC/RS variant), and Google obviously have the Colossus infrastructure that allows variable encodings, I'd be surprised if AWS is still triple-replicated everywhere.

Yes, and S3 is multiply redundant and is designed to survive a total AZ failure. So your data is physically replicated into at least 2 different AZs and might be multiply-redundant within them. They also provide a crazy SLA for data integrity, meaning that data must never be lost.

S3 also has a reduced redundancy tier and infrequent access tiers that are quite a bit cheaper.

It _is_ expensive, but once you crunch all the numbers, it's actually not unreasonable. I'd argue that using the real S3 is overkill for most scenarios that don't need infinite scalability.

GCS / Azure can survive a total AZ failure too, think of 3 x Replicas as RAID-1, whereas Erasure Coding is more like RAID-6. Only it's actually more resilient:

Let's say you have 10 data blocks, and you have 4 parity blocks. You can now lose 4 servers containing a block and still be able to repair the data, whereas in 3 x Replica you can only lose 2, and have to store everything 300% of size, instead of only 140%.

And yes, it is unreasonable how much they charge for both storage and inter-az bandwidth.

The problem with erasure coding is that if a disk fails, you suddenly need to read 3-5x more data to reconstruct the missing data from parity blocks. This is especially problematic if your replicas are split across zones. The inter-zonal bandwidth is large, but not infinite.

So I'm pretty sure that you need to have at least 2 full copies in different AZs, and then likely at least some additional redundancy within a single AZ (in the form of erasure codes or a full mirror).

So that's at least 3-4x the amount of data. 1Tb of NVMe SSD capacity is around $200 and with 3x redundancy that's $600, or about 2 years of AWS S3 storage. As I said, it's expensive but not unreasonable.

That's what Locally Recoverable Codes (including Pyramid Codes) are designed to address: you can repair a missing block using only a subset of the blocks, which you can make sure are placed in just one zone, eliminating the cross-az bandwidth requirements. Sure, if you lose multiple blocks, you are going to end up needing that extra bandwidth, but the chances of having two blocks offline at once is very low -- if it's just for a failure rather than extended maintenance, then you have probably already repaired the first block by the time the second fails.

In fact, the RS configuration is often in the >50 data blocks and >10 parity blocks range (albeit with an LRC/nested RS config) for object stores because it's more important to have that recoverability than repair efficiency. While one large provider I worked at did have a system whereby they did effectively have two copies of the RS-encoded data (so that 130% turned into 260%) across two AZs, they were actively in the process of swapping to the blocks being evenly distributed across the AZs, near-halving the total required disk space.

As I said before, most object storage is not on SSD, it's on hard disk: it's 20% of the price per TB, and most objects are read very infrequently. I can promise you that they're not paying $200 for 1TB of SSD either... I realise prices are higher than sensible in the last 6 months, but it was fairly easy to pick up SSDs for under $50/TB at retail pricing (and hard disks for under $10/TB) only a year ago.

You think that you're complaining about IAM, but really you're complaining about the web ui. I rarely use the web console, I use terraform or the cli. I'd you're vibe coding your infra with Claude, point it at the cli / terraform. Skip the ui.

With terraform you get the amazing experience of having to iterate, one at a time, through the five hundred and thirty seven new permissions you need to grant having decided that a lambda configuration needs to be ever so slightly different than it was yesterday, because there's no documentation linking terraform creation of resources and the IAM permissions required to successfully make the AWS API call behind the scenes. Or those for updating a resource, which are different, so you get to do it all again tomorrow. Or deleting - different again. Fun for the day after.

You could probably open the developer tools, find the console elements and extract the data from there to get around copy/paste limitations. I’m not familiar with the AWS console but let’s say it’s an input, select it in the dev tools and then in the dev tools console do $0.value

Agreed on IAM, and TFAs comments that once you see the horrendous complexity in IAM you start seeing it everywhere else in AWS as well. And with IAM, after all the effort you've put in, you can never really tell what is and isn't enabled. If you run your own server you can check permissions, run access-control audit scripts, and so on, and say with a pretty good level of confidence that X is possible and Y isn't. With IAM it's more like "I'm pretty sure I figured out the right silly-walk for X, but I have no way to tell what else might be enabled".

AWS: I came, I saw, I threw up in my mouth a little, I left.

I think the big problems with amazon IAM is not that it’s inherently complex, it’s that every team in AWS came up with their own way to define permissions and the calls these allow you to make. So the API Gateway set of permissions uses a completely different method for no discernable reason.

"who fought against Kubernetes adoption because it was "too complex", only to slowly reinvent Kubernetes badly,"

If you are dynamically scaling a set of web services sure. The problem is that people use k8s for running batch pipelines and streaming analytic services and a bunch of other things too. And k8s is terrible at doing those things and entirely too complex. And if you don't have to scale your web services very often, then k8s is a waste in that case too. Its a right tool for the job and k8s's job isn't deploying to the cloud, its dynamically scaling a website.