Pretty much every SV god has a reputation for being incredible jerks: Jobs, Gates, Zuckerberg, Musk, Ellison
So, maybe the rule isn't if you're a jerk you're a genius, but instead, if you're a genius, be a jerk. You just probably aren't a genius.
I wonder if it's the amount of ineptitude or not being on the same page that they may experience on a daily basis - even if it's 1 of 100s of people they interact with? I can see that being very trying when you literally have hundreads of things you need to make sure get done daily.
You could add Torvalds to that list.
If you really are a genius, you're a huge pain to work with, but at least you're worth something.
If you're not, you just look like an ass.