Absolutely not. Anywhere where accuracy, precision and safety matters, throwing LLMs in the mix is irresponsible IMHO or being too optimistic or possibly not understanding how these giant arrays of floating point numbers work or just hoping for the best.

Similarly, LLMs used for SQL generation meant for business analytics is also a critical area where if numbers are wrong, it might lead to a business going bankrupt.

For Prototype, fun exercise, sure go all in.

First of all its pretty awesome your permissions are very tight. You are definitely on the other side of the spectrum compared to the rest. I get it that there is a lot of skepticism because of people hyping LLM's so indeed for now we use it as Copilot and not the driver. Hopefully you can agree though its pretty random that we are still manually creating IAM policies and need to get accustomed with the thousands of different permissions :)

We are actively working on reining in permissions, I would not call them "tight". It's just not a top 3 priority, though that is likely changing with the upcoming SOC2 efforts. I still don't see us reaching for LLMs to help us here.

I'm not saying don't use them, just use them as an extra pair of eyes, mostly to catch errors rather than to drive and architect

> get it that there is a lot of skepticism because of people hyping LLM's

The skepticism is not from the hype, it's from experiencing LLM output personally. They are fine if the output can be fuzzy, like a blog post or a function signature, not so much if there is a specific and fragile target.

To add a plus one here, as soon as I learned there's LLMs involved this became a non starter to me. I'd rather have less granular policies than risk some LLM doing something crazy.

I can justify to management that we have limited time for IAM and something was missed that we can fix / create tests / scans for after an incident. It's harder to explain that we chose a vendor that uses a non deterministic tool that can hallucinate for one of the most core security pieces of the puzzle.

what kind of application changes are you thinking it would equire?

my policies are definitely too broad, but feels like I should be able to tighten them up without changing code. (just potentially breaking things if I get it wrong and go too tight).

Some scenarios

1. The application has to start using credentials for the first time, or consume them a different way. For example, stop consuming an environment variable and rely on a service account.

2. You have to change ops to support new workflows. Often you have to put approval workflows in place because fewer people can do things and you want only the machines touching production

3. You have to change human behaviors and habits (this is the real hard one). I've had to revert changes because the increased security blocked developers and they don't have time to adapt for the next deadline.

4. Getting parity in local development workflows is also challenging. How and where do you match vs except from IAM parity?

5. Should I give the current server access to a particular cloud service/resource or break out that particular function into a lambda and minimize the permissions there? You have to think through the implications of a breach and how/where you want to limit the blast radius.

6. This is probably obvious, but implementing application level controls, like API endpoint permissioning. IAM is not limited to cloud infra

The open-source project is a CLI you can put into your CI/CD so i think a pretty neat workflow where there should be less friction considering DevOps/security don't need to ping-pong on permissions.

when you keep telling, you ain't selling

ask questions to deepen your understanding

> ...ping-pong...

It was a scheduling problem rather than a decision problem. The impact radius is always more than you anticipate