Your points about index funds getting worse as they get bigger would apply to Vanguard too. And the whole idea that risk can be measured & mitigated in the first place requires predicting the future. Bankers in 2008 believed their portfolios were not overly risky either. Measuring risk is almost a contradiction, risk is the residual uncertainty that can't be measured after measuring/predicting everything else.
Overall, you are describing the modern economic portfolio theory approach to investing. Yes, its backed by some prestigious people but no one can predict how it will do going forwards. Its important to remember that uncertainty. In 20 years we might see inflation has swallowed up the returns of those investors and only people who invested in stock X made any money. We might see the reverse. Who knows?
Those FDA trials are expensive. Those companies have many experts in the field that believe those trials have a reasonable chance of success. Otherwise, they would not proceed with them. Even the FDA itself believes those trials have a good chance of success. There are plenty of Wall Street analysts who are also evaluating these trials. Regular Joe is not going to outsmart all those people.
However, you are right that the /r/personalfinance mantra of "S&P index or bust" is not actually good. In particular, as index funds become more popular in aggregate, they become worse investments, because they put continuous "buy pressure" on a select few stocks. Furthermore, the S&P isn't really diversified in that it only contains big cap stocks, afaik has no foreign stocks either (although of course many S&P businesses are global enterprises), and only has low exposure to industries like biotech.
Also /r/pf does a poor job talking about non-stock investments. Sometimes a better investment might be to start your own business, purchase real estate, or bonds. If you have time you can research growth stocks (e.g. pharmaceutical companies undergoing FDA trials) and pick the promising ones. Even learning a skill is often the best investment possible for most people - /r/pf constantly says that you can use about 4% of your wealth as yearly income, so you'd think there would be more people trying to get better jobs, since that makes a $20,000/year raise approx. equivalent to $500,000.