The answer to this is really simple: the people running the space. Run you own space, have your own CoC. Done.
If you can't get enough people to stand up and say "we're running a conference for tough as nails abrasive people, make sure you come in armored clothing", that might be a description of a problem.
The point is: In the end none of these group has a right to redefine the CoCs. They've joined a space which has been built by the "weirdo nerds" for their purposes and has the CoCs those deemed fair. Joining this space was no problem, the "weirdo nerds" welcomed them (or didn't bother to say: GO AWAY, both interpretations are acceptable), but now they try to dominate this space by abusing their skills in the "game of social skills" and rewrite the spaces CoC to one they like more and that's neither fair nor right.