Where possible I've started adopting the "run it and see" or audit2allow approach (there are awesome tools that can do this for AWS IAM perms too), but then before applying the policy, somebody needs to put a quick line beside each permission that explains why. If the answer is "I don't know" and the permission is simple/low-risk then maybe let it go. If it's a high-risk permission then don't do it without answers.
That formula is the only pragmatic one I've seen. Not perfect but sometimes perfect is the enemy of good.
This. Add enough hurdles, and people will
a) spend all the energy they're willing to spend on your process, hate it, and as a result never do anything except what you force them to, and in particular, never voluntarily reduce permissions unless forced (because they might need them later and then it'll be a pain). They'll also see you as an enemy, not a partner, which is not the place a security team should be in.
b) optimize for not what is best, but what is least affected by your wall of process, even if it's less secure (e.g. because it's a legacy system that you didn't get around to locking down yet).
c) outsource to less secure vendors, and get it approved because management knows that getting it deployed internally would take forever due to the process
d) in the most extreme case, set up uncontrolled shadow IT with zero security controls and hide it from you - and because they already spent all their energy dealing with "perfect", they don't want to hear the word "security" ever again, and the security posture of their shadow IT shows this.
Using a test AWS environment to stage and then checking CloudTrail to see what was actually called would be a step forward. Having software to extract it would be even better.
If I have the permission, I can tell its value in a few minutes but instead I have spend hours doing due diligence to try to justify my answer and still am not confident asking.
I usually end up getting what I need in the end so the admins don't see it as a big issue, but it is a subtle thing that can kills productivity for anything related to AWS for everyone else.
Not really easy to argue with the machine.
Unfortunately for development it is better to get tight integration of dev and ops so you could solve it by discussion and cooperation. Not sure if you can build such teams that often but that would be great.
Now as a consultant working with many large customers, “shipping fast” is nowhere near as important as security.
Did you use a managed service?
Or did you build it slowly and carefully on AWS?
I'm currently stuck between the two options. Managed service seems the easiest way to be HIPAA compliant but I'd rather we managed our own infrastructure on AWS since it gives us more flexibility for stuff like blue green deploys and it would be cheaper.
I didn’t know anything about AWS back then, they hired an MSP who was just a bunch of old school netops people who knew how to click around on the console and gave us a bunch of VMs.
Long story short, I studied for one AWS certification so I could talk the talk. I learned both all of the things that I could have taken advantage of and saw how much they were making and that changed my whole m.o. and decided to get some experience with AWS and go into consulting.
Next company I went to, the founders outsourced everything technically to an outsourcing company - software and infrastructure and they treated AWS as an overpriced colo. Everything was in one account and everyone had access to it. At first, they were just aggregating publicly available information about doctors for hospitals so it wasn’t a big deal.
They brought a new CTO in and started bringing development in house. I led the charge to first separate out the environment to different accounts, establish a sane CI/CD process and then lock down who had access to prod.
Of course they had secret access keys in config files everywhere. We had to audit the code to make sure that no code was using keys. Locally, every SDK can automatically retrieve the keys from your global config file (that’s nowhere near your git repo) and on AWS it gets permissions based on the attached roles.
Then of course we had to lock down roles. But we couldn’t have the granular permissions we needed because even though they had lots of microservices (we sold access to our APIs to businesses). They were all on two “pet” EC2 instances.
Next step was to move the .Net Core APIs to Docker/Fargate and further restrict the attached roles to those.
Finally, we had to audit all of our AWS dependencies and add encryption where necessary and then sign a BAA with AWS and bring in auditors.
By the time I left a month ago, we could pass the needed certifications and expand our offerings.
It took a lot of upskilling, hiring an internal ops person (I’m a developer who knows AWS) instead of depending heavily on the MSP.
I left for greener pastures - I’m a consultant with AWS.
How do you test a policy?
aws-cdk is their infra-as-code product and it's extremely valuable for iam alone.
You can say things like: my_lambda.grantRead(s3_bucket)
And it figures out the least privileges necessary to make all of that work.
Plus it's real code, not some annoying DSL, which means you can easily abstract other iam permissions out. I have a fairly tight lambda policy that I reuse in all sorts of places, and it's as easy to use as the above snippet.
There's this though: https://www.hashicorp.com/blog/cdk-for-terraform-enabling-py...
You get the best security when everyone is involved in the security design from the "ground up", but quite often there's not enough communication between the people developing some application, whose work may be valued by the number of features they ship and "velocity", and the operations side of things whose work is to provide the infrastructure for running the software, and to keep it running. At worst you just get some 3rd-party consultant to set up a thing somehow and then afterwards have to reverse-engineer it to figure out what the hell they did and how to prevent it from going up in flames immediately.
The most recent one I lived through a few months ago was when compliance just all of a sudden decided to wholsesale enforce a bunch or org-wide settings changes to every GitHub repo in the company, and it caused several outages and a huge amount of unplanned triage work as the settings were very sensitive for a bunch of continuous integration systems and jobs.
This was at a Fortune 500 company with a big, well-staffed compliance team. They had to roll back their changes and delay the new settings by several months because only through breaking production did they realize their proposes settings workflow was not feasible given in-house system requirements.
And of course, no apologies at all.
This is pretty run of the mill. I’ve seen the same thing from compliance and security teams in a few other large, “household name” tech companies, and also in a few mid-range startups.
Compliance teams number one MO is to blame product teams for not partnering with them, but it’s the compliance teams who refuse to do the partnering.
This is exactly backwards. Product devs need to reach out to security early in the design phase. There’s no way for a separate security org to understand the app or use case after the fact.
If you want to do $newthing your product dev management needs to involve security, finance, compliance, legal, etc. That’s their job. Developers don’t get to ignore all the normal business constraints the real world offers.
Building within constraints is what engineering is all about.
Reiterating what was mentioned in the thread - the best way to avoid this wildcard situation and make it easier for developers is to use Policy Sentry[0]
Thought I’d mention this for those who read the title and the comments instead of clicking on the tools. This will solve most of your problems with writing IAM policies for machine roles.
As far as adapters go, I know you can get SQL, Kubernetes, Terraform, Kafka, Envoy, s3 (via Minio), EC2/ECS/Lambda (linux). that would cover most use cases I think.
There's an incredibly broad set of permissions (at the cloud or OS level). Any app / tool may be written to use any subset of those. And what it uses is rarely documented (because developers don't see IAM security as a primary feature, outside of apps intended for use in regulated environments).
Without automation, this thus requires continual reverse engineering, which is never a healthy, sane long-term solution.
This should be fixed on the product / app side, where folks are much better placed to dump "I need this, and only this" in machine-readable form.
1) put ec2 servers I can't properly IAM lock down in dedicated accounts separated from all other things
2) don't let users create the things.. let users take actions that result in what they want by a programmed set of commands that is peer reviewed (e.g. spin up an EC2 instance with specified config/user data by an MFA'ed credential.)
3) user accounts are created manually (we have <10 employees with AWS access), but their user accounts can only do two things: a) massage their own account basics (rotate password and keys), and b) assume role. User account has MFA at setup and can't be removed (only changed). The roles they some are change controlled and checked regularly.
A LOT, like every damn security talk I've ever been to, says to watch cloud trail and enforce bad action by writing reactive scripts. We enforce before the action is taken, essentially. This works for better for EC2, which, in my opinion, is HORRENDOUS for least privilege.
Serverless items, while sometimes requiring more permissions than I'd like or expect, is far _less_ common to throw a requirement for "Create" on resource '*'..
Lastly, I give developers a playground to learn in that is entirely disconnected from user data. It has some mocked false user data and structures, and it's own isolated domain so the infrastructure can be explored
Whether this is "bad" depends on what you're targeting... it's bad for security, but good for getting things done. And from an economic standpoint, right now, unfortunately the "good" approach is often the "bad" one.
The $80M file may be less than the cost of doing it right. And until that changes, "good" managers will incentivize the bad approach.
The reality - everyone finds it MUCH quicker to give broad admin rights out otherwise.
One good thing - accounts - you can create an account, give admin to the consultant / outsourced IT group, still bill to org, they can do what they need without endless hassle of IAM. Anyone else using this - it's a pretty rough hammer but seems to work ok so people can get solutions spun up with some efficiency.
https://github.com/puresec/serverless-puresec-cli/
There is little reason that the friction of generating least privilege IAM policies can't be reduced. The same goes for deployment Roles.
At least, if you're very good and don't mind being perceived as a roadblock, you try to understand things. If you're more typical, you just find the most direct route from logged error to added permission (audit2allow approach). And if you're bad, which is also not uncommon, you just give it the broadest permissions possible and call it a day.
With respect to IAM in particular, I'm finding in the Lambda world that some seemingly straightforward functions wind up needing some sort of access to all kind of other AWS services; these services each have their own funky permissions structures and attendant quirks. Each one is a temptation to the IAM admin to just throw their hands in the air and put a wildcard on it.