I think at least one of his "autobiographies" is actually a bunch of his stories recollected by a third party, which doesn't help with that impression.
I somewhat did as well, but I imagine when you're as bright as him it's just something that's going to happen.
Then again, I probably would be too, if I was Richard Feynman.