I pair on all production code at work with only two other guys. I've worked with them for the last year. Together, any combination of the three of us is easily twice as effective as the fastest in our team. Something about the rhythm of the session, alternating roles, support when tackling boring parts, and the camaraderie frees us up to just get stuff done.
But, we work in a very complex domain that, a year in and many seminars by product later, we only barely are starting to grasp, with a large difficult to grasp system, sometimes solving problems just outside our comfort zone. It's not just CRUD and forms. So, maybe pairing is the four wheel drive of the programming world: uses more gas on the highway, but depending on your terrain, it might be the most fuel efficient way to get across a mountain.
Or: I've tried Vim and my writing/editing speed halved. Or: I tried APL but it took me half a day to write one line of code. Or: I tried playing guitar and it sounded horrible.
I get the feeling that maybe the outcome would be different if you'd try doing it for a while longer. No guarantees, though.
I think the trick is to use it when developers feel necessary to stay productive. No point having someone slogging away at something they find difficult and frustrating if a second pair of eyes and maybe some more specific knowledge of the area can help.
What bothered me about how I saw pairing used was that people seemed to make blanket assumptions about it's benefits. Many times I saw people pairing up on trivially easy tasks. Seemed to me that pairing was a lot like everything else, it can be done well or poorly.
Pairing should be a naturally occurring process IMO; ie I don't know how to best accomplish a task or 'story', so I ask a team member w/expertise or experience to help point me in the right direction. If I need help beyond that, it becomes a pairing/knowledge-transfer exercise. I came to refer to it as "informal pairing". In general I tended to gravitate toward pairing on the exceptionally difficult tasks or ones that would have far reaching design implications.
To elaborate, I've tried pair programming myself and it was completely inefficient when we tried it. I'm not going to dismiss it entirely though, perhaps we approached it the wrong way. Personally I just need a bit of space before I can start focusing in-depth about certain problems.
This is also why I like to be well-prepared before attending team design decisions, because coming up with good ideas "right there and then" is difficult for me.