Which is fine if you do everything locally (because you control that configuration), but not having escape really sucks when you are on servers a lot and you can't necessarily go changing vim settings around.

It's also difficult when using programs that emulate Vim's keybindings, because they tend to only use ESC by default.

C-[ is available by default; it's the actual control sequence sent by escape in many terminals.