These are the questions that periodic pairing answers with the lowest possible negative impact to members time.
Now, if you get into an environment that is 100% pairing all the time, or two people at the keyboard taking turns with an egg timer, etc...that can be a huge pain and I'd totally understand not wanting any part of it.
Watching somebody code while you drink a cup of coffee at your desk and offering the occasional "you know, if you do such-n-such it will be a little cleaner" or "I think there's a function that already does what you're writing"...is pretty low-impact.
EDIT: I read through your comment history and see that you were with Pivotal. IIRC they are a pairing all the time shop (I seem to remember seeing a presentation at some point). I can completely understand how that would suck.