Some people might do or were something that I won't like. I have learned to not let that influence me in way, and be comfortably ambivalent.
I especially felt bad after I came to know that Richard Feynman slept with many of his friends' wives and tried to sleep with even more of them. He burnt a lot of bridges this way. One Physicist had told me that Feynman hit on his wife twice, and when discouraged by the wife, he let it go as a perfect gentleman.
Despite this, Feynman remains one of my most revered icons and his picture still hangs on my wall.
I don't overlook what he did or try to find some naive justification. I know he was this way and that does not make his mind less fascinating to me.
Also in some blogs and articles. I cannot find these now.
Having a picture of von Neumann is in no way a red flag as much as having a picture of, or the name of a particular past US President would.
If you're going to indict people for having immoral sex, you'll have to indict probably 90%. If you include trying to have immoral sex, how many would pass your test?
I know the wives were equally guilty. He seduced but never ever forced to anyone's knowledge.
He often committed these acts of adultery when he was welcomed as a guest at someone's home.
Having adventures on the bed with the wives of the people who invited you in their homes- pretty low to me.
The women were adult and consenting, but it is still betraying the trust of people.
If all these were to happen with the consent of the husbands, although it would be extremely weird and unnatural, not to mention- immoral, by the standard of the stark majority of any society, I wouldn't have given it any thought at all.
I don't think you would have been disappointed :-)
Documentary (with English subtitles ) https://m.youtube.com/watch?v=RuwnywSpl40
Documentary (in French) https://m.youtube.com/watch?v=QQui90ZGobo
That being said, there are quite a few historical figures (Einstein, Penrose, Hawking, Ramanujan, Bertrand Russel, Grothendieck) that are nowhere as morally questionable as many, though they are all flawed people.
On a purely intellectual level I doubt that you would be disappointed though they are all human in the end.
What kind of conversations could we hope to have with someone like von Neumann or Turing?
Early in the Cold War -- while the US had a nuclear monopoly -- Von Neumann was a strong proponent of an all-out, surprise first strike against the USSR by the US. "With the Russians it is not a question of whether but of when. If you say why not bomb them tomorrow, I say why not today? If you say today at 5 o'clock, I say why not one o'clock?"