I wonder if it's the amount of ineptitude or not being on the same page that they may experience on a daily basis - even if it's 1 of 100s of people they interact with? I can see that being very trying when you literally have hundreads of things you need to make sure get done daily.
You could add Torvalds to that list.
So, maybe the rule isn't if you're a jerk you're a genius, but instead, if you're a genius, be a jerk. You just probably aren't a genius.