'Aloof' is a subjective term and you are entitled to judge him that way if you want to.
I'll assume, 'aloofness' to mean that his thinking was confined to his work and that he lived by himself without much contact with the outside world (a la Pearlman).
The reality may have been the exact opposite of this, if you look at his biography. My main source is Walter Isaacson's book. Einstein was a humanist who cared much for the well being of humans. He was a social person by all accounts, forming warm relationships with people from all walks of life. As a youth, he was a keen student of music and philosophy, as evinced by his deep involvement in a study group with a couple of his friends. His personal relationships and tribulations (e.g. divorcing Mileva Maric and marrying his cousin) seem no more deviant than that of many normal people. He was a committed pacifist who campaigned for many issues by using his standing in society as an intellectual. He supported the Zionist cause, decried racism and bigotry, etc. He wrote many essays and expressed very clear and reasonable opinions on a wide range of topics ranging from the American society to nuclear weapons.
Compare that to Pearlman declining the Fields Medal and Newton dabbling with alchemy and occultism, which are both representative of fairly weird behavior. To reiterate, 'weird' by societal norms, which individuals may or may not be expected to uphold (unlike, laws).
"The terms "chemia" and "alchemia" were used as synonyms in the early modern period, and the differences between alchemy, chemistry and small-scale assaying and metallurgy were not as neat as in the present day. There were important overlaps between practitioners, and trying to classify them into alchemists, chemists and craftsmen is anachronistic."
As for Witten, I just looked up a video of him on YouTube. One of the top comments, unfortunately very inflammatory:
"If Witten wasn't a mathematician he would have been a serial killer."
I supposed this is referring to his unorthodox mannerisms of speech.
(The video: https://www.youtube.com/watch?v=Y6b3DjOnv3I)
http://content.time.com/time/magazine/article/0,9171,993017,...