Exchanges and other prominent users would probably also have to agree to the change. Things might get bad if there was a split between miners and everyone else because the miners could perform 51% attacks on the other fork, though.