I index for a very good reason, some of the smartest people I know chose to work on Wall Street, and they're putting in 60+ hours a week working on this stuff. If you have tens of thousands of people like that, I don't care how smart you are you just aren't going to beat them. Better to insure you are at least average (which net fees probably puts you in the top 30%) than to chase being in the top 5% and failing miserably.
For a specific example, look at how the "hobbyists" are beating the "professional" analysts at predicting Apple's quarterly performance, time and time again.
http://tech.fortune.cnn.com/2010/04/20/apples-blow-out-quart...
The point is that AAPL is now only second to XOM in terms of market cap, and is not exactly a tiny blip on the radar. It is a stock that is being watched by literally millions. If these "professional" analysts have so many resources at their disposal, have the inside track on confidential information and are operating at such an uneven playing field as the parent postulates, then why would they consistently be beaten by the "hobbyists"?
In fact, I'd argue the information asymmetry tilts in favor of individuals, considering that analysts publish their forecast. I wouldn't be shocked if these guys use "analyst expectations" as an input.
Anyway, this is bad news for the analyst, because the one resource they don't have is time to focus on one company. Hobbyists, however, can focus on Apple all they want. So I'm not surprised that Apple fans nail their earnings forecast.
If no one was trying to beat the market, it would be easy to beat the market. But since so many people are busy trying, it's silly to try to compete unless you want to take it as seriously as the professionals do and get plugged in the way they are.
The "random walk" hypothesis says you can beat the professionals. About half the time. By flipping a coin. But the professionals are insiders, and as well-regulated as the stock market is, I would assume that an outsider playing against insiders in a 50/50 game will not quite win his full 50% share.
The hope that a person can price stocks better than the market as a whole rests on a couple of possibilities:
1. Beating full-time professional analysts at analysis in your spare time, because you're just so bad-ass.
2. Identifying a price that is heavily influenced by ignorant people and beating them by just being reasonably well-informed -- but this is not an original idea and hence is a special case of #1.
3. Finding a niche that you know intimately for some reason, and in which you have no professional competition. Oh, wait, this is just #2 again. Or maybe you found a niche small enough that it doesn't get a lot of attention from professional analysts, and you can make a little bit of money at it. That might work!
4. Being the first one to take into account a new idea or source of information. This one will make you wildly rich, and it's fun to dream, but while you're looking for that brilliant idea, there's no reason to risk real money on your ideas-in-progress.