News for hackers. Stuff that matters.
Re:AWS Networking Concepts in a Diagram by nprateem on Saturday July 22, 2023 @05:06AM (#36824648)
Why is debugging IAM policies and auth issues such complete dogshit on AWS? Seriously, if you want to be a competitor, focus on this.

I've wasted hours trying to work out why I get an unauthorised error, and the official docs say to manually pick your way through the 8 or so different policies that might apply (SCPs, IAM, resource policies, etc). Yeah right.

When you eventually find the page showing how to use Athena with Cloudtrail, you discover half the requests are inexplicably missing. And even though error messages include a request ID, you can't easily query for them, if it's even possible. Maybe it is and I just haven't discovered the right athena incantation. They should make it easy to just query by request ID and get clear messages telling you what policy denied the request.

It's a total train wreck from start to finish. I guess by keeping such a shitshow they sell more support contracts though.

Re:AWS Networking Concepts in a Diagram by trevorsullivan on Saturday July 22, 2023 @05:09AM (#36824664)
Pretty much everything in AWS APIs is a trainwreck. They mindless wrap objects other objects, for seemingly no reason. The customer bears the burden of learning and implementing the extra logic for complex data structures. It's incredibly annoying. They need to pair up UX designers with engineers, but they don't.
Re:AWS Networking Concepts in a Diagram by benterix on Saturday July 22, 2023 @07:05AM (#36825227)
A cynic in me is happy about this state of affairs - the companies who happily joined the AWS bandwagon have to pay the price (including my salary). For my projects I almost always choose something else that offers better value, except in very rare cases when it makes sense.

A decade ago or so AWS was marketed as - to simplify - "fire all your sysadmins because we manage the systems for you" to the current state where good AWS DevOps engineers are expensive and not so easy to find and you pay through the nose if you really follow AWS recommendations, especially for larger setups.

Re:AWS Networking Concepts in a Diagram by nprateem on Saturday July 22, 2023 @08:17AM (#36825584)
Pretty tedious work though. I'd far rather feel I was actually doing useful work instead of dealing with this stupid crap.
Re:AWS Networking Concepts in a Diagram by steveBK123 on Saturday July 22, 2023 @08:20AM (#36825604)
Yeah this has been my observation.

CTOs bought the "fire all your sysadmins" literally, and now we have a 20% overhead on all app devs who are supposed to learn all this stuff.

It's crazy to me. It's literally the opposite of every management training / book about focussing on highest value work / where you add value / where you provide expertise.

Re:AWS Networking Concepts in a Diagram by renegade-otter on Saturday July 22, 2023 @09:49AM (#36826208)
20%? Seems like at least half of all the work has to do with infrastructure plumbing, now that people who are supposed to write product code are expected to be Cloud prodigies.
Re:AWS Networking Concepts in a Diagram by hooverd on Saturday July 22, 2023 @11:09AM (#36826876)
I hate CloudFormation I hate CloudFormation.

Unless you're using AWS as yet another VPS provider, creating a "cloudy" product requires way too much knowledge about AWS services as a developer.

Re:AWS Networking Concepts in a Diagram by ecshafer on Saturday July 22, 2023 @04:10PM (#36829387)
When I was at a company that went all in on AWS. I went from 100% of time working on product, to easy 75% of time working on AWS. Even though we had teams who just set up infra on aws for us to use and it was supposed to be "so easy". There was far more time setting up AWS infra than writing code.
Re:AWS Networking Concepts in a Diagram by steveBK123 on Saturday July 22, 2023 @04:32PM (#36829605)
Yes, and management discounts that .. if you make me, a product expert, working on application software.. deal with this stuff, I am going to do the worst, most rapid, half-assed, security and everything else be damned job.

I have clients, product, scrum masters, tech leads, and others breathing down my neck on feature delivery, and you're asking me to figure this alphabet soup out, lol.. don't expect happy results dudes.

I've been a domain expert in a field for 20 years, what value do I bring to the client in dealing with this alphabet soup of AWS networking/security? Either it works and client gets what they want, or it doesn't and they don't.

This stuff is a full time job and should be respected as such. If you think you can push it all down on app dev, you are going to have unhappy results.

Re:AWS Networking Concepts in a Diagram by anyoneamous on Saturday July 22, 2023 @09:57AM (#36826268)
I never saw it marketed that way - but even so, anyone stupid enough to fall for it deserves every NAT gateway charge they've had since.
Re:AWS Networking Concepts in a Diagram by 2-718-281-828 on Saturday July 22, 2023 @10:52AM (#36826717)
would you mind elaborating on the nat gateway part? are you insinuating that nat gateway as mean to provide oneway internet access for instances in private subnets are misguided or suboptimal?
Re:AWS Networking Concepts in a Diagram by hooverd on Saturday July 22, 2023 @11:04AM (#36826829)
Managed NAT Gateways are one of the more expensive services in the known universe.
Re:AWS Networking Concepts in a Diagram by 2-718-281-828 on Saturday July 22, 2023 @11:16AM (#36826940)
what would you use instead?
Re:AWS Networking Concepts in a Diagram by benterix on Saturday July 22, 2023 @11:40AM (#36827161)
AWS NAT Gateway pricing is so outrageous that smart people came up with a peculiar hybrid:

https://www.lastweekinaws.com/blog/an-alternat-future-we-now...

I wouldn't use it in production (some companies do, for internal things) but the very fact it exists speaks volumes.

Re:AWS Networking Concepts in a Diagram by berkle4455 on Saturday July 22, 2023 @01:20PM (#36827981)
a tiny ec2 box that provides NAT for about $3/mo.
Re:AWS Networking Concepts in a Diagram by devonkim on Saturday July 22, 2023 @08:05AM (#36825516)
FWIW, more debugging information in an IAM failure is another round of information available to me as an attacker. If such a service existed available to the customer this also becomes potentially available to an attacker. In fact, it may wind up violating a few companies’ security and compliance policies.

However, I think that customers should have the option once they sign some stuff acknowledging that they may be really, really screwed similar to Bad Things done as a root user in AWS.

Given the distributed nature of so many requests in AWS I’d imagine that a GraphQL-like setup would be conceptually compatible with such a system. It’s not terribly far from tracing systems like OTEL

Re:AWS Networking Concepts in a Diagram by nprateem on Saturday July 22, 2023 @08:19AM (#36825594)
For some APIs they return encrypted payloads with actually useful debugging info in, but there seem to be plenty where you just end up in a black hole of "computer says no"
Re:AWS Networking Concepts in a Diagram by JoBrad on Saturday July 22, 2023 @08:51AM (#36825798)
They could put better information in Cloudtrail, at least.
Re:AWS Networking Concepts in a Diagram by booi on Saturday July 22, 2023 @03:06PM (#36828860)
They do! Most all IAM errors will list the actor, the resource, and the missing permission
Re:AWS Networking Concepts in a Diagram by bob1029 on Saturday July 22, 2023 @10:12AM (#36826389)
> Why is debugging IAM policies and auth issues such complete dogshit on AWS? Seriously, if you want to be a competitor, focus on this.

System managed identities + AAD is an absolute joy in my experience on Azure. I can grant a function app access to a SQL database by simply using the literal name of the function app (qualified by the object's identifier if aliased). No passwords, certificates, etc. involved anymore. It literally just works.

Combine their auth package with things like application insights, and you are not really ever stuck thinking "WTF happened", assuming you have enough awareness to look at those logs.

> I guess by keeping such a shitshow they sell more support contracts though.

This is also a problem with Azure. You really have to know where all of the mines are located before you start walking through their field or you will wind up in a similar state of contorted frustration. I'd argue that Microsoft leaves substantially better hints regarding where mines might be, but they still conveniently leave out enough hints such that only a wizard with 2 decades of experience can pierce the final veil of bullshit and avoid a 5-10x multiple on cost & complexity.

Re:AWS Networking Concepts in a Diagram by berkle4455 on Saturday July 22, 2023 @11:38AM (#36827137)
Chinese hacking groups say the exact same thing about Azure, access is so easy!
Re:AWS Networking Concepts in a Diagram by cameronh90 on Saturday July 22, 2023 @11:41AM (#36827167)
Unfortunately with Azure, while authentication may be more obvious, 0x80003007 ERR_UNKNOWN type things aren't.

Also, what's up with the way almost every any change in Azure takes some random period, sometimes hours, to take effect and there's not necessarily any indication that anything is happening.

Re:AWS Networking Concepts in a Diagram by scubbo on Saturday July 22, 2023 @04:17PM (#36829467)
> I can grant a function app access to a SQL database by simply using the literal name of the function app (qualified by the object's identifier if aliased). No passwords, certificates, etc. involved anymore. It literally just works

I can't comment on the general correctness of your comparison (never having used Azure), but this particular one is flawed, since the same is true in CDK. That said, there's enough arcane weirdness in AWS' configuration requirements that I suspect your point is _broadly_ correct.

Re:AWS Networking Concepts in a Diagram by jiggawatts on Saturday July 22, 2023 @05:58PM (#36830369)
> It literally just works

In only a few toy scenarios!

For example:

Azure SQL Database, Managed Instance, and SQL 2022 in a VM all support AAD authentication.

Okay.

SQL 2022 supports only user accounts and not managed identities.

SQL Managed Instance requires an additional global AAD permission button to be pressed post-deployment or it won’t work. This can’t be templated.

The first two have AAD permissions controlled via their template (ARM API) — but only exactly one full admin group. All other permissions must be assigned via T-SQL. Again, can’t be templated via ARM. (Microsoft has better data plane RBAC for random open source products than their own flagship database!)

User access audit logs are easily collected for Azure SQL Database, needs manual T-SQL for SQL MI and is missing in action for SQL Server.

This is just SQL, which now at least supports user-assigned managed identities… if you include the latest SQL client which has a breaking namespace change and hence won’t work with third-party code. It also drags in 500 MB of dependencies.

Lightweight functions, you were saying? Hmmm…

Anyway, I got fed up with the whole thing when I tried hooking up a Storage Account in the same way and discovered that it didn’t support UMSI in connection strings — you need two code paths for dev-time and runtime!

Up until recently the whole thing was an unstable mess. Authentication would take 5 seconds in some scenarios and then wouldn’t cache the tickets properly.

It turns out that every Azure team is implementing the whole thing separately! It’s not at all like traditional AD where the platform provides the authentication functions in a uniform way and apps can just pick up the ambient tokens.

No, in this shiny new world every service does RBAC differently, connection strings differently, auditing differently, caching, config, and on and on.

All of this within 100% Microsoft products and services.

It’s a poorly thought out mess.

Re:AWS Networking Concepts in a Diagram by bob1029 on Saturday July 22, 2023 @08:33PM (#36831394)
> In only a few toy scenarios!

I am not sure I understand. The "toys" you describe are all we need to get the job done. Azure SQL Hyperscale, Azure Functions C#/v4, AAD, another region for DR (SQL geo replica + Function App x2) and that's about it... Why do we want to make this hard?

If you want to try to force the square peg through the round hole, I get it. It's going to be a clusterfuck as you describe. But, I decided we were going to get completely trashed on the Microsoft kool-aid so we don't have to worry about 3rd party clients and other non-happy-path integrations. I click 1 big dumb button in my Visual Studio client and some time later my app is running on a live URL completely managed by Microsoft. This is a pretty good life. I don't have patience for things like I used to. Leadership & investors are not giving me much to work with these days.

Re:AWS Networking Concepts in a Diagram by jiggawatts on Saturday July 22, 2023 @08:43PM (#36831449)
> square peg through the round hole ... Why do we want to make this hard?

I work in a job where I don't get to make product decisions, I just have to "make it work" on the cloud.

Despite this, the only applications I've worked on in the last three years have all been ASP.NET web apps with SQL databases. Even with this pretty vanilla combo, I found hundreds of bugs or interoperability issues in Azure. These aren't esoteric things, they're not round pegs and square holes. These are Microsoft-shaped pegs that ought to fit into Microsoft-shaped holes.

Things get messy the instant you step off the "tech demo MVP" happy path.

Like I said: Third-party code that talks to SQL, Storage Accounts, or SQL databases other than the one of their two Platform-as-a-Service offerings!

Even if you stick to just functions and Azure SQL Databases, there are still limitations and issues.

Templating is painful or impossible. Data-plane RBAC is missing in action. Etc...

If you're a solo developer working a tiny app deployed with click-ops to a single region, then I suppose it is "fine".

For everyone else, it's not so fine...

Re:AWS Networking Concepts in a Diagram by evulhotdog on Saturday July 22, 2023 @10:59AM (#36826782)
Those request IDs can be used if you escalate to AWS support, they can point you in the right direction. It’s generally not a fast process, but it’s something.
Re:AWS Networking Concepts in a Diagram by nprateem on Saturday July 22, 2023 @01:11PM (#36827905)
Exactly. The tools exist, they just don't make them available to customers.
Re:AWS Networking Concepts in a Diagram by awinter-py on Saturday July 22, 2023 @11:46AM (#36827217)
ugh access + least privilege on AWS + GCP are really hard

not made easier by the fact that debugging service accounts can sometimes require rebooting boxes or clusters

'cloud traceroute' to discover where exactly the problem is would be amazing

(in fairness there are 'least privilege' tools I haven't tried -- listed below and I recall reading about others)

1. iam access analyzer https://aws.amazon.com/blogs/security/iam-access-analyzer-ma...

2. airiam https://github.com/bridgecrewio/AirIAM

3. policy simulator https://cloud.google.com/policy-intelligence/docs/iam-simula...

Re:AWS Networking Concepts in a Diagram by adduc on Saturday July 22, 2023 @04:30PM (#36829595)
Cloudtrail writes to s3 at intervals, which means it make take a few minutes for the log entry related to your request ID to show.

I've found Athena doesn't appear to search through all log entries (possible user error on my part); I've been downloading the cloudtrail logs directly from S3 and grepping through the logs directly to find the relevant entries to figure out the permissions needed.

If you need to decode an encrypted error message, use `aws sts decode-authorization-message --encoded-message $MESSAGE`. It'll return a JSON string. I typically use `aws --profile limbledevadmin sts decode-authorization-message --output text --encoded-message $MSG | jq .context.action` to extract the needed permission.