If he is pompous, he is so because he wants to drive home points which people think are unconventional and idiosyncratic. But in fact the points are real. I just wish people could understand him more, without any biases whatsoever.
Other people often called "arrogant" - Paul Graham, Steve Jobs, etc.
His writing style in general is meandering and lacks focus. The book I read had a handful of insights amid pointless narration. I read books like The Black Swan to learn new ideas, not to learn how great the author is.
I found that "When Genius Failed" gives a great story-telling journalistic perspective on some of the same incidents Taleb talks about in his book while "The Drunkard's Walk" provides a good mathematical intuition (sans math sadly) behind seemingly rare events.