There is nothing wrong with sex parties, nor drug use. But a lot of these VC-adjacent parties have reports of strong power imbalance- “young female founder seeking funds, wealthy VC seeking partygoers”. That is the issue with them.

(Like those described in the removed link)

Altman is a married gay man, so his involvement in them seem… less likely.