If there is actual data to suggest who the drug works on best, and empirical tests to determine who that is, it can absolutely be tested within those limits and approved for use on that population.
What happens is the drug companies decide at stage III what patients to let into the trial. What they are trying to avoid is anything that will limit the market size once approved. This means they try to skate as close as they can to the effectiveness limit and still get the drug approved.
For example, if they have a new cancer drug that they know only works on cancers with certain mutations, what they will try to do is to test the drug in a broad population of patients including those without the mutation. Since for the approval process it does not really matter how effective the drug is (as long as overall it is statically significant) and that you can only market to the population you tested in, you will want to shove in as many “filler patients" as you think the drug can support. Due to the difficulty of designing large trials they often get the filler number wrong. I don’t condone this activity, but this is the commercial reality.
More fundamentally the pharmaceutical industry is being crushed between two very powerful forces - the cost of developing new drugs continues to get more expensive every year (mostly thanks to regulations), while the real market for each drug shrinks as sub-populations are identified for each disease. Combine this with a refusal of society and insurance companies to pay ever increasing drug prices and the outlook for the industry is not great.
If you want to get a good idea of what really goes on in the pharam industry have a read of the “In The Pipeline” blog [1]. Most of the real value is in the comments.
1. I honestly don't give drug companies that much integrity.
2. Sadly, I feel they systematically lied to the FDA in order to get their product to market? No, I don't think the average pharmaceutical research scientist wanted to poison people, but I have horrid feeling a lot of them went along with the sharade in order to make a living? I picture too many scientists thinking, "Well it doesn't work as we thought, but I'm really not hurting anyone because the Plecebo Effect was huge? So, in order to keep my job, I will not object to these results/statistics?". I have found that when it comes down to a person's livelihood, life--money; people don't raise their hands? They go into this weird, socially acceptable denial?("Hay it's legal!") That's why when even here on HN, I make sure to read the blurry statements.(I'm usually disappointed, but I want to hear the dissenters, and critics.)
3. I'll admit I threw the baby out with the bath water years ago, and I don't have hard evidence (in front of me collated, and ready to provide objective examples) to back up my intuition, and assertions, but I don't think one has to look too far for examples?
4. I grew up not questioning scientific integrity. I thought these institutions/people were the pillars of honesty!; what I have learned in the last decade, specifically about the Pharmacutical Industry, hurt me on so many levels--I literally feel like crying sometimes.
5. My understanding of statistics is fuzzy. To those that need a refresher, I found two books helpful: How to lie with Statistics, by Darrell Huff., and The Comic Book Guide to Statistics--don't know the author.