Even though the pool operators have large discretion, the computational power that makes up their pools can be moved elsewhere... a little like a transferable proxy. Most smaller miners probably trust the pools who they've grown confident in... but a large enough controversy or campaign could conceivably shift the mining-power away from pool operators who make unpopular decisions.