Totally, I think if something is making you quit it must be pretty annoying - or you wouldn't risk trading the known with the unknown (which can very well end up being worse). I also wouldn't mention the real reason in a future job interview, if it sounds ridiculous.

That said, small things can look trivial from the outside and make your life a living hell at the same time.