One thing to consider: it’s clear he was very widely connected in American elite circles. It’s likely that almost all prominent people (in finance, business, the arts, science, government, etc) all came across him. We know he worked pretty hard to get intertwined with everyone he could.

And yet, although the list of his connections is large, it’s very far from most of these people. It seems plenty of people saw him for who he was and steered clear.