Usually, once it's become apparent that some people are getting fabulously wealthy by breaking away from the herd, jealousy and Dunning-Krueger take over and you get a large number of people that are suddenly true believers in active investing.
1. You may not even be able to acquire shares or to dispose of them (everyone is passive!)
2. Low liquidity + required multiyear holding times = dramatically increased risk
3. Businesses are affected by the markets their shares trade in. Earnings can be depressed by increased costs of capital, sentiment and anti-competitive pressures from shareholders who hold the entire market.
The third is mildly concerning. If a few companies are swimming in money, they may disturb every market around. Historically this don't last for long, every time the market gets like this, there comes a crisis and destroys those big players. Worth keeping an eye to a "this time is different" event.
Now, the second one is the real problem here.
The first can't happen because you can't answer what is the correct price of a stock. :) That's a bit glib, but the only reason passive investors are said to not correct is because there are plenty of active investors to move the prices of stocks.
So if you're actually trading rather then just collecting dividends (and even then), then whether a stock moves is based less and less on the actual position of the business.
Holding them is fine, but index funds are the exact same thing: they're an exercise in cheaply diversifying to the broadest possible scale, which is the only way the little guy beats the algorithmic traders.
That's the Castle-in-the-Air theory of investing, and it's been popular for much longer than even digital computers, let along HFT (Keynes is said to invest based on it, and as described the logic in his Beauty Contest analogy). It's why many thousands of investors over a century have been poring over stock price and volume charts, looking to predict where the other investors will put their money and beating them to it.
There are certainly algorithms nowadays doing this, but they are certainly not limit to HFTs, and have been around for much longer.
It may be contributing to greater wealth inequality, though: if you look at basically anyone who has become fabulously wealthy in the last couple decades, it's because they saw a profit opportunity that others were unwilling or unable to exploit. The primary form of unwillingness is "It's too risky..."