Yes, cultural fit is real, and it isn't always about white men drinking craft beer.
One of the downsides of pair programming is that it's great when you both have complimentary skillsets. It's less productive when no one knows what the heck is going on. In this case, the task was to benchmark how many writes one could make to Kafka using our monolith. They wanted to see at least 10K/writes per second. The only problem was no one knew how to use Kafka (well), including the person who was asking for it.
So I wrote some gnarly looking Bash code to fire up Kafka, Zookeeper, the app, etc. Some people wanted to use Docker, but I didn't see the value in using a slower VM w/ limited resources. I wanted to use as much CPU as possible. My pair didn't know Docker, Bash, or Kafka. I lost my pair somewhere around talking about having to clean up zombie processes.
It went downhill from there.
The only time I found it to be a liability was when I was pair programming. My pair couldn't read a shell script to save their life & had very little interest in learning. When I tried explaining it line by line, they stopped wanting to pair because I was apparently making them feel stupid. They actually reported me to management for being an arrogant mansplainer. I'm pretty sure that was the beginning of the company deciding I was not a culture fit :P I had to learn how to, and how not to, work with someone less experienced in a certain topic after that.