The way you describe it, a developer’s role is to point out stories and jointly perform the work the PM has allocated.
Instead, if those same developers understood the users, use cases and metrics and co-created the work the team does, the team would have 10 or mor engaged minds all working to make a better experience instead of the one super PM.
If the team's programmers are "product-minded" [1], then the PO will be getting lots of input from them, and in a healthy team will be going along with most of it - the PO is in a "first among equals" kind of position. If the programmers are not like that (junior, new to the domain, just not built that way), then the PO will make the product decisions without them (but still with the support of a product designer, hopefully!).
There's an excellent leadership principle in the UK armed forces (and i'm sure elsewhere) that a leader must consult their subordinates before making a decision, but that once a decision made, subordinates must go along with it. The PM - programmer relationship around prioritization is an example of that.
[1] https://blog.pragmaticengineer.com/the-product-minded-engine...
Also when you are doing the 101st CRUD type function in an app, you probably cannot be that enthusiastic about managing yourself either.
So like you said, devs learn to get better at forcing the PM's to split the stories into smaller chunks. Smaller stories are almost always easier to accurately point since it is always 2 (1 point for the story, 1 point for testing).
The numbers are not meaningless though. That is why velocity is an average of 3 weeks. A single week doesn't blow things out of the water and if it is a new project, the expectation is that we won't know where we are at all those first 3 weeks. It gives a built in ramp-up period for the project.
Agreed, the PM's role is SUPER important. For this reason, at Pivotal the PM's usually have a developer background.