I used to think I would want to be an angel investor at some point, but being an LP at YC (and maybe at a later stage VC, and get shares in great long term companies directly or via BRK.A, would really meet my goals.
I think it is really hard to become a limited partner in good VCs, unless you are godlike in some domain (which I guess you especially would be when you eventually exit) or were a founder of one of their successful companies.
I understand that Y Combinator only does token investment in terms of money, and gets a good share of the startup in exchange for its brand and the experience and connections of its founders within the VC world. Since the actual money outlay is low, it can afford to fund hundreds of startups.
I doubt that's something that the average "wealthy person" can do.