Unfortunately there is no per-site override. So you get to the occasional site that you don't want to zoom (like a game where the zoom gesture does something in-game) that is unusable with this global setting enabled :(