So if you want to beat the market, your chances are still better in the passive group...
Now, this doesn't take into account getting away with insider trading, of course. If you can get away with insider trading, then you can beat the market just fine.
More importantly, it also doesn't take into account that there are investment opportunities that are not as directly (i.e., without intermediary layers which extract some of the returns) available to passive investors, so that the passive/act groups aren't splitting the same universe of alternatives. (E.g., passively investing in any given index fund is on average better than actively investing restricted to stocks in that index, and you can probably even extend that to passively investing in index funds in generally being better than actively investing in investments that directly held by any combination of index funds, but not every investment vehicle in the market is directly held by index funds.)
Is this really true? I think it's only true if you replace average with median (e.g. in [1,2,3,10] only 10 is above the average).
> [I]f all of the "passive" half have a different random subset, then about half will by definition do "above average", and the other will do "below average". While in the "active group", less than half will do "above average" because they pay part of their gain to fund managers.
So, if all the passive investors have pick a random "large enough" subset of stocks, one would expect the distribution to approach the normal distribution, with roughly half to either side of the arithmetic mean.
Similarly, as you point out, on the "active" side, one might see more outliers - and perhaps the selection isn't really uniformly random.
So, it would probably be a stretch to claim that half will be above average "by definition", even if one includes the expected distribution, while note perhaps entirely wrong.
Also, as I understand it, Berkshire gets to implement management changes (if it deems them necessary) as it wishes, since it owns whole companies, and so the improvements they see could also come from better management of a business rather than picking better businesses. So arguments about average or above average stock picking ability seem less relevant here. (It could well be that Buffett has average management-picking ability, but is one of a relatively small number of market participants who both can and do exercise it, thereby getting an outsize result.)
ISTR a study a few years ago that said the only way you can know for certain that a fund manager is above average is on something like 30+ years of data. There's too much random variation.
I think it was the same article that said the baseball season would need to be several hundred games before you could definitively choose the best team in a given season.
Perfect example: Buffett has always touted value as a great investment strategy, but he completely missed the cocktail of global trade, speed, and scale that have driven Amazon, Google, Uber, Netflix, Apple, etc. to grow revenue and profits so rapidly.
Looking back, I think this shift into a world where it's possible for a company to go from zero to hundreds of millions/yr in net revenue over a span of 4-5 years (Uber) is really going to be seen as some kind of historical turning point in business, on par with the invention of the printing press, or electricity. Companies used to take a century to accomplish growth that's now possible in less than a decade.
Which is all to say, over a long enough timespan, you're talking about a pretty deep and complex philosophical question, what it means to be a "good investor" in the face of so much change, risk, etc. To be honest, I'm not sure it's something that's knowable at all.
"Properly measured, the average actively managed dollar must underperform the average passively managed dollar, net of costs. Empirical analyses that appear to refute this principle are guilty of improper measurement."
When someone picks active management over passive, they are really betting that they can beat, and hence are not, average. It's difficult for people to swallow their hubris, even when simple arithmetic shows them that they are (collectively) wrong.
http://www.cfapubs.org/doi/pdf/10.2469/faj.v47.n1.7