I might not be surprised this particular drug has this particular effect, but you have to replicate it rigorously, and then show the mechanism isn't due to something prosaic (like people feeling better because their GI tracts are in better states).
1) Someone defined "statistically significant" as 5%, and sometimes it's too high and it's too easy to get an interesting result caused by a fluke, specially if there are too many groups trying different things. (As an extreme case, in particle physics they use 5 sigmas to avoid fase discoveries.)I think we agree about this.
2) In this case, reading the paper linked by criticas below in https://news.ycombinator.com/item?id=28745673 , the error bars in Figure 1 are unbelievable small. For example the second points looks like 12.0±.2 vs 13.0±.2 (I can't find he exact data, so I'm just looking at the graphic and guessing.) And they claim p<0.01 T get this level of concentration with only 20 persons, the raw data mus be very concentrated around 12 and 13, but I expect that the number of remembered word to be very noisy and have a wide distribution.
> 44 healthy volunteers aged 18-36 participated in the trial. 23 were given prucalopride, and 21 were given a placebo.
> “Participants who had taken prucalopride for 6 days performed much better than those receiving placebo on the memory test; the prucalopride group identified 81% of previously viewed images versus 76% in the placebo group. Statistical tests indicate that this was a fairly large effect – such an obvious cognitive improvement with the drug was a surprise to us”.
What is the p value of this? There are missing data in the press article, does someone have more information?
My guess is that the variation in the number of images you can recall is like 20% (or more), and ~25 subjects reduce it like sqrt(25), so a 5% of difference between the trial and control group is not very significant (p<.04999).