That's also a likely reason why family office holdings often look very odd as they often only provide a hedge in case the main source of income dries up. And therefore, they cannot have any connection with it. Pure performance on a standalone basis is often only second priority.
Why? A couple of billions should still be a tiny drop if compared to the market cap of S&P 500. What do those active portfolio managers provide to you?
I think the S&P500 is an entirely different beast to what it was 20 or even 10 years ago as a result of this.
If you were a family office your goal would be to get exposure to pre-IPO growth stage stuff as well as the public market.
With a couple billion you can get yourself some nice deals in a bunch of different asset classes. Also with that kind of money you can put a chunk of your money in illiquid investments that potentially could provide some great returns.
I don't think active management makes sense for the majority of folks, but billionaires are exactly the kind of people that it does make sense for. With a billion or two you can probably get yourself into some pretty good closed-end funds (if you're savvy about it) that will probably deliver better risk adjusted returns than vanilla passive.
Personally, if I had the money, I would go with 2 Sigma or AQR. Both of their past returns streams are stellar and uncorrelated with the markets.