Here you are asking the most important question.

Exchanges hold the most ETH. They are a central collusion point which could cause undue influence on the network.

At least it seems they do not own a lot of the existing Eth.

https://etherscan.io/accounts

Their actions would most likely be visible on the network because it is transparent. The community can then decide whether to activate a fork and punish the colluding nodes.

Who is the community and how do they decide to fork(eg: is there a vote?)?

Users, developers, other stakers.

Why would users and developers without stake get input? Doesn't that subvert the point of staking?

Users and the market decides on the chain, governance in Ethereum is based on social consensus not “one coin one vote.”

I still don't understand how consensus works off the chain. If 51% of the stackers just decide to run their own software and do their own thing. How does something off chain change that? How does something off chain penalize a majority of staked ETH?

If the 51% is a single staker or small group of colluding stakers attacking the chain, the rest of the users will probably not want to continue using that chain. The users can activate a soft fork, just by changing the rules of the code to burn the attacker’s funds, and running the updated client software. The remaining honest stakers can then continue to follow the head of the non-attacked chain, and the attacker would be the only one left on their chain.

If it's just off-chain social consensus that matters what purpose does the crypto even serve?

Most crypto including Bitcoin is governed off-chain. People come to consensus on a set of rules that allows them to create a public and permissionless BFT ledger. Attacking the ledger then costs millions or billions of dollars. If one dishonest node or a group of colluding actors decide to spend this much to amass majority control of the network, the rest of the network can decide to fork the protocol to burn their staked capital.

The main goal of crypto - the base token and protocol rewards from PoW or PoS - is to secure this network and keep it permissionless and decentralized.