> poor IAM policy and role modeling

This is a bad take. Making good IAM Roles and Policies is incredibly complicated if you have a complicated account. You WILL get it wrong. This becomes much more tractable if you have reasonable account boundaries between workloads.

If you insist on a single massive account you're fighting against the way that AWS designs the system, and you're gonna have a bad time.

Totally agree. Defense in depth, security in layers. You're not protecting against just the most elite hackers, you're protecting against mistakes. Mistakes and change are inevitable, they should be in the design.

I'm not even sure at what point account boundaries really have merit anymore past organization. Most things that are subject to account boundaries have really poorly crafted IAM roles or policies somewhere. You just have a single logical boundary in between the two talking things.

The way that GCP organizes various different projects is actually really nice. I haven't seen what this looks like at scale though.

> You have poor architecture, poor tagging, poor VPC design, poor IAM policy and role modelling or don't know what you are doing to start with.

I would flip it around and say that historically AWS has had extremely inconsistent architecture and IAM policy design that can make it very hard, sometimes impossible, to do it the "right way".

The nice thing about using separate accounts is you don't have to get into as many of the hairy weeds and the permissions you end up might end up being much simpler to create and then also to maintain down the road, since everything is isolated by default and then you allowlist only the things you need.

I don't see why you would frame this as "you fucked up" in your design.

Even if it only protects you from fuck ups, isn't that part of good design? People WILL make mistakes, limiting the blast radius of a fuck up is important.

Historically yes. Your job is to evolve this configuration as security controls improve. It's not a fire and forget process, it's continuous improvement.

Or you can just use multiple accounts, which makes things a whole bunch easier.

Frankly, AWS is just missing a level of abstraction here. Azure has resource groups, Gcloud has projects. An AWS account now is just used instead of those concepts, despite it being heavyweight and awkward to do so.

There's plenty of tools to automate the creation and management of new accounts. The biggest hurdle afaik is there's no automated way to delete an account

Azure also has higher-level subscriptions

It does, but account creation is kind of slow, and the whole control tower / SSO / etc. stuff is fairly janky. Clearly Amazon have been trying to make the account a more common level of isolation for some time and it's improving, but it's still not fantastic. Support also still has a minimum monthly pricing and isn't cross-account.

AWS recently added the organizations:CloseAccount API (albeit with some caveats discussed elsewhere in this comment tree).

We keep dev/staging/prod in different accounts, and we host some special environments for customers which is in separate accounts as well.

The only reason you want to do these things is for security and liability purposes. They are not an advantage for organizing things, but rather they protect against accidental resource deletion and unauthorized / unintended access.

It’s cumbersome to manage many accounts, but that’s a feature, not a bug: it’s intended to be difficult to switch between accounts, so that you can’t accidentally fat finger the wrong button.

They don't really though because in the several systems I've seen configured like that, the same staff have access to all accounts. Accidental deletion and unauthorized access should not be a possibility via policy rather than isolation, because if the former isn't done then the latter will not help. If the client demands it, sure, but charge them for the inconvenience.

The main risk is hitting resource limits which is why I would and do give development teams their own account to cause mayhem in.

The places I've worked just control that with SSO and groups in their auth system. Much easier to make a group in your auth software than try to create multiple permission boundaries in the same AWS account. It also makes cost allocation much easier (you don't need complicated tagging policies everywhere)

Security in layers.

Infrastructure as code.

Easier for you, easier for the black hatters.

Disagree. Shared accounts may work for a smaller org, but not for larger ones.

If you need multiple accounts for isolation or data protection you can accomplish the same thing with scoped IAM policies, so you can only touch things with the correct tag or name in path, this is hard to maintain and confusing for developers.

When you are large enough there's no avoiding multiple accounts as you run into API limits and hard quotas. The added benefit of having an account per system is that you can have simple IAM policies, essentially : compared to scoped described above.

Networking is simple with shared VPCs, even with hundreds or thousands of accounts. Unclear how this affects egress at all. Don't know about billing but we have an Org with enterprise support, guess that helps. The awslogs cli tool makes it easy to extract logs via cli, just login to each account in its own terminal window or use profiles.

I worked with one of those tag based accounts a few years ago. Incredibly frustrating because it was locked down to the point you couldn't see the IAM policies and figure out what tags or metadata you were missing.

You’re absolutely right, I’ve fucked up plenty. That’s why I believe so strongly in making the right thing the easy thing. I think I’m doomed if it’s critical for tags to be perfect because they always drift, doomed if IAM policies must be least-privilege because sometimes that’s impossible, and doomed if I can’t adapt what I built yesterday to what my business needs today.

PS good job partitioning your EKS clusters, even within one account. That’ll save you some sleep one day, I’m sure.

> Every other reason here is because you fucked up.

No. Just No.

This is a guardrail against fucking up. And everybody needs more of those.

The only thing missing that would make this the best thing ever, is AWS having a limit on how much cost an account can generate. But they love making money of people's mistakes, so you can never prevent the infinite downside.

Having things “in a VPC” doesn’t help unless you are just using AWS as a glorified Colo. Once you start using AWS native services - many of which aren’t in a VPC, VPC isolation doesn’t help.

Each AWS account also has its own service limits and quotas. One out of control Lambda in the dev “environment” can impact production.

And you don’t have to open support tickets manually. There are APIs to request service limits and you can monitor the progress programmatically.

I run into "IT professionals" all the time who don't even realize you can run services without a VPC.

Besides, every time you try to create a “least privileged role” to run infrastructure as code, that role has so many privileges it’s easy for mistakes to cause mistakes in your production “environment” that you meant to only affect your dev “environment”.

And I know I’m probably preaching to the choir. But one of the misconceptions I have to constantly fight is “we run our Lambdas in a VPC for security reasons”. (Lambdas are never “run in” a customer VPC)

Yes, I tried arguing with someone that it was actually more secure to run your Lambda without associating it with a VPC. If it doesn't need access to the VPC resources, it shouldn't be "run in" one. What can be more secure than having absolutely no access?

cfn_nag has this as a default security rule. So there’s one place to lay blame.

There's definitely a place for these counterarguments, but I have had Amazon's own consultants tell me that quota issues are exactly the reason they think you should architect to span many Amazon accounts if you're big. There are some quotas they simply can't raise as high as you want them on a per-account basis.

The main counter to this is account limits. For example we had our staging and live environments in one account. We load tested on staging (which has its own VPC) expecting it not to affect production. But because we hit account limits in the process of load testing, production Lambdas stopped responding as we exceeded the account limits.

We now have a Control Tower setup to ensure that anything we do to staging can never take down production, even if it's an account limit breach.

> Every other reason here is because you fucked u

Ok...

> My favourite thing doing is spending 2 days opening support tickets in 10 different accounts to get a limit raised and then tracking the state of all the tickets and limit changes...

Oh. So, you fucked up too? Because all that should be automated.

> I disagree with this perspective. You should have multiple accounts but only if your organisation requires it for isolation or data protection reasons and only enough to perform the task.

I'm not sure where you got your take from. The blog post you're commenting explicitly makes the case for multiple AWS accounts with isolation in mind. In fact, the blog post barely mentions anything beyond isolation. How can you claim you agree and disagree at the same time?

> You have poor architecture, poor tagging, poor VPC design, poor IAM policy and role modelling or don't know what you are doing to start with.

Yeah. So you are saying it’s a great idea for 80% of all companies?

> My favourite thing doing is spending 2 days opening support tickets in 10 different accounts to get a limit raised and then tracking the state of all the tickets and limit changes

I agree this sucks balls though.

Then don’t do that.

There are APIs to both list all of the accounts in the organization and request service limits. It’s a three line bash script.

Let me take that back, you have to assume the role in each account using either the autogenerated Organization role or Control Tower role.

> There are APIs to both list all of the accounts in the organization and request service limits. It’s a three line bash script.

You assume I have access to all that. Then there’s the situations in which you request a limit higher than what they’ll automatically give you, and you end up running through 10 support requests :/

If you don’t feel like writing the three lines, ask your TAM.

Surely you aren’t administering your AWS accounts clicking around in the console all day? You are responsible for opening tickets for accounts but don’t have permissions to use the CLI within CloudShell?

lol, I can sense someone who have never got their account blocked for some arbitrary reason.

Just imagine you woke up and your AWS account is blocked and all your infra is shut off. What's your plan? Blame "poor design" or policy?

If you want to build a _reliable_ infrastructure, the only rule is to avoid singe point of failure. Account is one of those.

No worries, it will come with experience.

Are you suggesting having 2+ copies of your infra so that you can withstand losing one of them? I've never heard anybody propose this approach before.

> If you want to build a _reliable_ infrastructure, the only rule is to avoid singe point of failure. Account is one of those.

I'm not sure which sense of the word "reliable" you're using here. Usually durability / continuity of business conversations focus on having backups in other accounts (on same or different IaaS providers).

Availability conversations are usually centered around multi-region within a single IaaS provider, and that can be in a single account. Sure, you could build it to use multiple accounts, and you might pat yourself on the back if it ever actually saves you, but how long would it take to build that and what else could you have done with that time?

> I've never heard anybody propose this approach before.

Some people do, some people just talk. Noone proposed don't mean noone did.

> Usually durability / continuity of business conversations focus on having backups in other accounts

That's quite poor quality of conversations then. Continuity of business is much wider than having backups.

> how long would it take to build that and what else could you have done with that time?

Lesser than you may have expect. What could you have done with your time is certainly a good question, the best asked when your infra is down because cloud provider blocked your account. As I mentioned in original comment, if you ignore probability of that, then you are just not experienced enough.

> My favourite thing doing is spending 2 days opening support tickets in 10 different accounts to get a limit raised and then tracking the state of all the tickets and limit changes...

Counter intuitively, a single or small N of accounts makes this more likely. With many, have fun, take the safety bars off, it is contained.

If there's one thing that is true in this world, it's that you can't rely on people not fucking up.

All you can do is try making fuckups difficult by figuring out how to implement systems where doing the right thing is the path of least resistance. You won't stop fuckups, but a system where people will do the right thing 90% of the time instead of 50% is so much better.

This is wrong advice.

You need multiple accounts, and that has always been the best practice: [1] and [2].

Where are you keeping your Cloudtrail logs? On a Bucket on the same account!!? That is the first thing that will be wiped out, in case of a security compromise of the role, or user, that set it up.

You can also, and should, use AWS Organizations for consolidated billing [3].

Even without AWS Organizations, with a single account, particularly at an Enterprise level, you are just a step away from loosing everything. And the reason why you might loose everything, has nothing to do, with the best practice of only using the root user, (with MFA) only for the very few required tasks that force you to use the root user. [4]

One possible scenario, is if for example, the persons that have acess to the root user credentials, ( and there always be some...) are going rouge, blackmailed, or getting compromised in another way. All gone, including your remote backups in other regions...

S3 Object Lock [5] will mitigate the risks of a compromised root user but don't use only one account. Have more than one, even for your personal projects.

[1] - "Benefits of using multiple AWS accounts" - https://docs.aws.amazon.com/whitepapers/latest/organizing-yo...

[2] - "Organizing Your AWS Environment Using Multiple Accounts" - https://docs.aws.amazon.com/whitepapers/latest/organizing-yo...

[3] - "Consolidated billing for AWS Organizations" - https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2...

[4] - "Tasks that require root user credentials" - https://docs.aws.amazon.com/accounts/latest/reference/root-u...

[5] - "Protecting data with Amazon S3 Object Lock" - https://aws.amazon.com/pt/blogs/storage/protecting-data-with...

One other motivation would be a fully functional backup so your product keeps running when they arbitrarily disable your first account (bonus if it's on a whole other provider).

Is billing aggregation a problem and do you need to open up 10 different support tickets if all the accounts are part of an AWS Organization?

As for keeping the number of accounts down, I've seen that blow up significantly if you have someone break into an account or have a disgruntled employee that can get into a single account and wreak massive damage. Or if your software runs in multiple regions and you need to meet compliance requirements like GDPR, etc.

Actually upgrading your Support plan is one of the very, very few things you still need to break into the root of each account to do. However, if you’re big enough you just sign an EDP contract that forces all your accounts onto Enterprise Support anyway.

This papercut has, as of last week, finally been fixed: https://aws.amazon.com/about-aws/whats-new/2022/09/aws-updat...

Actually, as of Friday, that’s no longer true. \o/

<https://aws.amazon.com/about-aws/whats-new/2022/09/aws-updat...>

None of these concerns require multiple accounts. An account is a container for resources and resources do have tangible isolation between them if configured properly and properly delegated credentials should be issued to staff which are specific and limited in capability. Same with assumed roles. Same with VPC configurations.

If you didn't do that, you fucked up. Adding more accounts doesn't guarantee that you didn't fuck up.

And yes you need to open 10 tickets, even if you have enough spend to have a direct line to AWS internal staff with multiple enterprise accounts...

- get all of the account numbers in your org

- for each account number

   - aws service-quotas request-service-quota-increase