"Is it fellow academics? Or university administrators? Or government agencies that control grants?"

Yes.

Fellow Academics: The people who will be discussing your tenure case, writing letters of support, nominating you for prizes, etc.

University Administrators: The people who are the final word in your tenure case, and who do things like evaluate how well your department/college/etc. is performing.

Grants: Be they your fellow academics in the form of reviewers, program officers, etc., the prestige of your publications will likely matter to them.

Publications are the unit of currency in academia at the moment, and the incentive and evaluation structures at almost every step are oriented around that. Going your own way is a laudable step, perhaps, but largely the luxury of established researchers who have already had their prestige publications, or those willing to take the potential hit to their careers for the principled stand.

Yep: every group cares because others care and because they don't want to make things easier for others later (that would not be fair). Academics care because they need to get grants and promotions, granting agencies value the papers because they need to justify their grants to the public/Congress/academics, etc.

If you could get them to all drop caring at once, that would work. However, if any critical mass of any majority of groups sticks with it, those who try to ignore the high-profile publication wheel will just get squeeeeezed out. Simple game theory.

Exactly. Which is why big labs going pre-print/OA is meaningful, but the heads of those big labs saying "As Chair of my Department, OA publications will be positively considered in tenure, hiring and promotion decisions" would probably be a bigger deal.

I should also mention that my graduate program (top-tier in its field) unofficially required all students to publish in a top-tier journal to be allowed to graduate with a PhD. It was unofficial, but enforced by all the professor/advisors. If you didn't publish in a high enough journal, you'd end up in 7-8th year with a long thesis and have to leave with a master's degree.

If you go the principal investigator route (professor or scientist at national lab), then your boss and granting agencies are the ones that care about your pubs. It's critical to obtaining tenure (if you don't, you're kicked out of the university) and grants.