So, how does the FDA deal with this issue today? Suppose the scientists wanted to market (some flavor) of jelly beans as a drug to cure acne (instead of to cause it), and they test 20 flavors and get evidence at p < 0.05 that white jelly beans cure acne. Or 1000 flavors and evidence at p < 0.001.
Does the FDA look for a causal mechanism that would make the drug effective? Or are the companies afraid to market things that they genuinely suspect are ineffective because they'll eventually get sued? Or is the FDA's effectiveness standard so strong that you just can't cost-effectively game it like this today? (The last possibility seems a little bit unlikely.)
As far as I know nobody has ever been sued over a drug that didn’t work. From a purely legal perspective you would better selling a sugar pill than a real drug that worked and caused 1 in 100,000 people to drop dead.
A better approach would be to not have any effectiveness trials at all and just collect good data on all patients using the drug. If a drug really worked it would become obvious the more patients used the drug, while if it was worthless this would also become obvious. A new drug would start out unknown with only the most desperate trying it and overtime we would learn more and more about its effects and side-effects.
This bayesian approach was not possible in the past, but with modern technology it is now. We would get better drugs with better known side-effects at far lower costs.