This is a good point. However in practice we wind up with a number of gatekeepers between the developer and the actual users. These gatekeepers are both on the customer side and the developer side. For example on the developer side, sales people typically do not like being disintermediated since it reduces their relevance. On the customer side, managers often do not want outside developers bothering their workers.

As a developer I've had plenty of pointless and unpleasant customer interactions thrust upon me. It's not a good thing.

I have always appreciated product managers who effectively intermediated customers, had deep domain knowledge, managed stakeholder expectations, learned how to interpret what they said (and ignore what isnt relevant), asked the right questions, prioritize effectively and output to me a nice set of, de-noised, ordered, precise answers to the question "what do I do next?"

It's a rare skill. It boosts my productivity immensely when done right. It removes cognitive load from the developer.

Even when POs make mistakes and misinterpret requirements I still appreciate that being somebody else's problem. It quells my anxiety immensely and lets me focus on developing the thing right without having to worry about whether Im developing the right thing.

Because that isn't something their management levels will even consider a possibility, given security concerns, politics, and whatever might come up if the project goes south and a lawsuit is bound to happen.

These agile ideas only work in an ideal world of "we are all friends here".

I'm sure this is true of some companies, but I've definitely seen this done well in a few places. Often software agencies.