Intestinal drug shown to boost memory and cognition

Re:Intestinal drug shown to boost memory and cognition by derbOac on Monday October 04, 2021 @10:16AM (#28747028)

Not to criticize this finding, but the way to think of these things is in terms of p values corrected for multiple testing over the sample of studies conducted every year. That is, if there's M studies every year, some fraction of those will be significant at whatever significance level just by chance, even in the absence of any questionable research practices. Such is the reason for replication.

I might not be surprised this particular drug has this particular effect, but you have to replicate it rigorously, and then show the mechanism isn't due to something prosaic (like people feeling better because their GI tracts are in better states).

[ Parent | on HN ]

Re:Intestinal drug shown to boost memory and cognition by gus_massa on Monday October 04, 2021 @09:42PM (#28754602)

There are two problems:

1) Someone defined "statistically significant" as 5%, and sometimes it's too high and it's too easy to get an interesting result caused by a fluke, specially if there are too many groups trying different things. (As an extreme case, in particle physics they use 5 sigmas to avoid fase discoveries.)I think we agree about this.

2) In this case, reading the paper linked by criticas below in https://news.ycombinator.com/item?id=28745673 , the error bars in Figure 1 are unbelievable small. For example the second points looks like 12.0±.2 vs 13.0±.2 (I can't find he exact data, so I'm just looking at the graphic and guessing.) And they claim p<0.01 T get this level of concentration with only 20 persons, the raw data mus be very concentrated around 12 and 13, but I expect that the number of remembered word to be very noisy and have a wide distribution.

[ Parent | on HN ]