These are the questions that periodic pairing answers with the lowest possible negative impact to members time.
Now, if you get into an environment that is 100% pairing all the time, or two people at the keyboard taking turns with an egg timer, etc...that can be a huge pain and I'd totally understand not wanting any part of it.
Watching somebody code while you drink a cup of coffee at your desk and offering the occasional "you know, if you do such-n-such it will be a little cleaner" or "I think there's a function that already does what you're writing"...is pretty low-impact.
EDIT: I read through your comment history and see that you were with Pivotal. IIRC they are a pairing all the time shop (I seem to remember seeing a presentation at some point). I can completely understand how that would suck.
If pairing periodically is enough of a burden on you that it would cause you to interview elsewhere, it really seems like you're overthinking the impact.