The real concern is that this philosophy has terrifying possibilities (even if they're in the fringe), and being associated with powerful and wealthy people like Musk threatens to validate and realize those possibilities.