I have a lot of experience with XP. In my experience, XP succeeds because of its extremeness, not despite it. It's not for everybody, but for organizations and teams such as Pivotal who are willing to understand and embrace it, it works very very well.
So I challenge your "given that it's surely not because of all this" comment. Do you have any evidence? Or are you just saying, "I don't like this, so it can't possibly work?"
The kind of "pairing all the time" kind of XP that is being preached is similar for programmers. It also works for repetitive work - building yet another web application - at scale, not at scale, not much of a difference. But programmers often get into the job because it is an inherently creative field. There is nothing like this kind of XP to suck it all away and leave them disillusioned about the craft until they escape and find a product role or even a consulting company with more empathy and kindness.
However, we had a few guys on our team that really didn't like it. They left and I don't think less of them in any way. It's a different way to do things.
As for the "repetitive work" aspect: the most successful XP project I did was a collaborative sketching application using pen input with automatic shape recognition: somewhere around 2000 (when did MS put out those pen based computers??? -- We did it for their unveiling).
The problem though is the Agile Industrial Complex and "thought leaders" who have pushed this as a religion on the industry. I enjoy pairing, and the excitement of two minds working at the same problem in tandem is an experience to be relished. But give it to an Agile snake-oil salesman, most of whom gravitated away from programming a long time ago in favor of making money training others in how to program, and they'll wield it as a weapon by prescribing the only true way to work.
XP (in its traditional sense) doesn't scale past 8 developers anyway. Anybody selling you XP for a team of 500 is someone to be wary of ;-)
There are pains and giving folks the opportunity for change/rotation is refreshing.
Software is a team sport, its a weak-link game where we must communicate and empathize in order to see clearly (often past our own egos) in order to pull one-another up.
I was fortunate that good pairing hygiene was a focal point. Without that it can be disastrous mentally and ruin the experience for everyone. It turned into a highly collaborative sport at that point.
I am a hugger naturally but when I run into those pairs I went through the trenches with, it is a brother/sister hug. Best time of my life.
There’s also a survivorship / selection bias inherent to companies with extreme views like this. Either you both (a) agree to cheerlead & display positive affection for the religion and (b) display effectiveness in the role, or else you will quit or get fired or self-select to never work there in the first place.
So in a large part, it’s literally unknowable to what degree success is in spite of vs because of these extreme practices. There can’t be evidence about it because by definition there is the confounding factor that the employees whose productivity you measure are only from one specific sub-population (likes extreme view & delivers good software). You never get to measure what people with diverse opinions about tbe philosophy would do in the role, and you never get to measure what less successful developers would have done in the extreme philosophy.
The same effect happens with extreme coaching philosophy in sports, where again after you apply the filter that the players are selected to both be very good and accepting of the philosophy, there’s no way to measure the counterfactual possibilities, so it reduces to religion.
Pivotal can deliver things to customers in spite of all this (given that it’s surely not because of all this), which sort of makes any advice published out of there fundamentally irrelevant for people working pretty much anywhere else.