I used to think this, then I worked on a trading system in an investment bank. Don't underestimate how quickly and easily you can learn and exceed people who should know what they're doing, given sufficient motivation.
The fact that you saw some people you perceived to be clowns once, does not demonstrate the market is anyone's for the taking.
What Buffet does and what HFT do are completely different. The reason why Buffet has to change how he trades is because of HFT. If Buffet puts in an order for 100,000 shares of Coke (KO) while it's at $40 a share, he would not stand a chance. HFT bots would swarm in and scoop every last share of KO that's available and then sell it back for incrementally more. All within a fraction of fraction of a fraction of a second.
I agree there must be some niches are that are amenable to exploitation by small independent traders (perhaps anything to do with nanocap stocks).
I have not had success finding niches where I thought I would be successful, but I could very well be insufficiently clever.
But trading highly liquid equities on a multi-day timescale (the strategy proposed here) is certainly not prohibitively hostile territory for large, sophisticated buy side operations.
By arguing for ETFs, you are implicitely assuming that there is not much to gain by doing your own research because markets are efficient and have already priced in everything.
By arguing that most people underperform the market, you are implicitely assuming that it is easy to underperform the market - something that should in fact be hard if markets were efficient and everything priced fairly.
Also, most people underperform the market because of fees. If you invest in a mutual fund with a 2% management fee, then the fund needs to outperform the index by 2% to breakeven. It needs to do significantly better than that if you're invested in a hedge fund with a typical 2-and-20 fee.
Edit: in particular, small-cap stocks tend to be inefficiently priced because it doesn't make sense for institutional investors to research them heavily since they cannot allocate a large percentage of funds to them without: 1) significantly disturbing the market price 2) in some cases owning a significant percentage of shares (5% or 10% I think) that requires filing with the SEC.
I think the argument is that many people (including fund managers) trade too actively, which generates costs that cause them to underperform an efficient market.
Just as a side note, I was watching the excellent ESPN series "30 for 30" and they were describing why so many athletes have gone bankrupt so quickly after retiring, even though they supposedly made millions when they were playing.
Your quote is the key. They had several financial planners and they said the same thing, "It's not cool for these guys to their money in an index fund and watch it grow over 20 years. They want bars, dance clubs, music studios, and other frivolous stuff. THAT'S why they broke."
The stories the athletes tell are pretty jaw dropping by the way: http://www.youtube.com/watch?v=TSOAwNSv8EM
Don't try to be clever. Just invest your money in a few different indexes and let it be.
Last year my account grew ~30%, it's increased 16.1% so far this year (the market isn't as crazy as it was in 2013 but it's not bad, either). You need to be disciplined enough to let your money sit and resist the temptation to do dumb things when the market dips.
It's not rocket science, the best way to fuck it all up is to try and manage your money yourself and play the market. Don't do that. There are thousands of bankers out there working 12-14 hours each day to take advantage of people who don't know what they are doing. Don't be a sucker.
We are in the midst of a 3 year old bull market. We are just barely off all time highs in the S&P and Dow. Nasdaq is also killing it -- even after some pullback in high beta (volatility) stocks.
Your returns the last 2 years are not typical of an average year. A return approaching double digits over a long trendline is not beyond reach, but it would be a mistake to bank on a 12% return IMHO, and even more of one to suggest other people could expect the same.
There are tons of calculators where you can play with average returns over time, check one out.
The ones that crashed and burned in the recession are the ones most likely to provide a high return in a bull market, there are plenty that were largely unaffected by the recession (mostly securities) but you won't benefit as much from a bull market with those.
I have a blend of different funds.
There is nothing special about Vanguard ETFs, and even if you feel you are diversified there's an old saying that in a severe downturn the correlation of everything goes to 1. So most likely, in a downturn, everything you hold will go down, even if you feel you're diversified. This of course is not a law, just a probability.
(And just as a helpful FYI, both stocks and bonds are securities)
Everyone looks smart during a bull run-up. That same SPY allocation would have been crushed in 2008-2009. On average, you'll get average results.
The CAGR of SPY for the past 100 or so years is about 6.5%, inflation adjusted. Good, but not 30% good.
I agree with everything you said above, including this. I want to add though, that there is effectively free money in the stock market. For example, just by buying a low-fee index fund (e.g. something from Vanguard), you're almost guaranteed to do better than most investors and probably better than nearly all speculators. (And there are other investment strategies that typically outperform the indices as well). I guess the reason people do poorly in the stock market is similar to why people start dumb startups that don't really have any hope of being profitable: The idea of rapidly creating an enormous amount of money for very little effort in a very short amount of time is much more appealing than making 12+% per year indefinitely, even though this strategy is much more likely to net you a higher return in the long-run... plus you actually have to save money if you want to invest this way :)
Edit: It's also interesting to learn about how some of the big quant trading firms started. D.E. Shaw, for example, originally had some bond trading algorithms they used. It was very profitable and the hours were short compared to the rest of the Wall Street/Finance world. Then they got greedy and tried some more aggressive strategies, blew up, and nearly lost the fund. Fortunately for them they seem to be doing much better now, though I'm not sure what their current strategy is.