This is a bad explanation of p values. Of the start:

> If you’re testing whether mean=0 and actually the truth is that mean=0.000000001, and if you have a large enough sample size, then YOU WILL GET A TINY P-VALUE.

Think about this for a second. Does it make sense that the results of my statistical test would change if I measured in terragrams vs. micrograms? No, it doesn't, because it's not the numerical difference between null hypothesis and sample mean, it's the difference relative to standard deviation.

If the effect size is 0.000000001 and the standard deviation is 0.0000000000000001 then this is an enormous, important effect. If the standard deviation is 0.1, then this is a really small and unimportant effect. It can still be observed correctly with a sufficient sample size though because you've stated it's a real effect to begin with.

You will absolutely not get a tiny p value just because your sample size is large. That is wrong. You will be able to detect very small effects, but if there is no difference, you are more likely to draw the correct conclusion with 1,000,000 samples than with 100 samples.

The point the author here is trying to convey but is not getting clearly, is that a large sample size can detect an effect that is practically useless and call it significant. p-values tell you if it seems probable, given your data, if there is a difference between the two datasets. They make no claims about whether the difference is material.

Example python code to do prove p values are not likelier to be smaller just due to large sample sizes, given no actual difference:

   import random
   from scipy.stats import ttest_ind

   num_tests = 10000
   res = []
   for test in range(num_tests):
      n = 100
      a = [random.random() for _ in range(n)]
      b = [random.random() for _ in range(n)]

      res = res + [ttest_ind(a,b)[1] <= 0.05]

   print(f"10,000 tests with n=200 {sum(res)/ num_tests}")

   num_tests = 1000
   res = []
   for test in range(num_tests):
      n = 100000
      a = [random.random() for _ in range(n)]
      b = [random.random() for _ in range(n)]

      res = res + [ttest_ind(a,b)[1] <= 0.05]

   print(f"10,000 tests with n=20,000 {sum(res)/ num_tests}")

This is a good explanation, thank you.

>You will be able to detect very small effects, but if there is no difference, you are more likely to draw the correct conclusion with 1,000,000 samples than with 100 samples.

I believe that this is to the tweet author's point where they stated that in the real-world a null hypothesis never exactly holds true. So you can detect even the tiniest of variance with a large enough sample.

>given no actual difference

I think the error here is that there is a priori knowledge of no difference, where the author is stating that in real-life scenarios there will always be some differences even if being too minute to be of practical significance. We can fabricate "no difference" in simulations but in real experimental design there will almost always be variance, even if it's just an artifact of the measurement process rather than being causal from the independent variable(s). Whether or not the differences statistically significant depends on the experiment design, to include sample size.

So while I understand your valid point I think the author's claim was more about the practical application of statistics rather than the mathematical precision as it relates with simulated examples. But I could be misinterpreting.

If you're performing a rigorous experiment, then you have a control group, and your null hypothesis is that the experimental group will be the same. In actuality you will find that many null hypotheses hold true and the experimental design has no effect on the outcome, at all. Of course in some fields, like psych, most everything has some effect. But it's absolutely not correct to blame p-values for helping you distinguish between no effect and astonishingly small effects. They're functioning correctly. That is a secondary problem to solve, usually by stating a minimum effect size.

Of far greater problem is False Discovery Rate related things. Where you test 20 different things at once, and by chance identify one of them as significant even though the true effect size is 0. This is another area where increasing your sample size can help avoid problems, but even still you need to acknowledge your tools are imperfect.

>In actuality you will find that many null hypotheses hold true

I'm assuming you mean within the confines of the experiment, correct? I agree. The tweet author was eluding to the fact that "IRL" the null hypothesis is almost never true at the population level. Meaning if you grab a large enough sample you will detect very, very small differences. (This was her Lucky Charms ~ blood type example in the tweet). I also agree with that. I don't think the two claims are mutually exclusive and the fact they can coexist is (I believe) precisely her point about sample size.

I mean there are many real world examples where the impact of A has no effect on B, obviously. The number I'm thinking of, has, truly, no effect on the time since you last blinked. No sample size will change that.

Lucky Charms does probably relate to blood type in some impossibly small way. It makes sense that a biological trait has some relation to dietary consumption. I don't think any sort of non-garbage tier journal with peer review would publish it, but good on p-values for helping us detect them though. Not bad on sample size for making it possible to discern this effect size with a high degree of confidence.

It's worth noting that we also have many other tools to help us. For example you can test, given an expected effect size and a sample size, what the probability is of getting a statistically significant result, or a non-significant result, or a significant result that erroneously goes in the wrong direction. Or what the range of likely true mean effect size is given a significant sample difference.

We want large samples. They enhance confidence in findings. The author's premise seems to be it's better not to know that small rocks exist if we're only looking for big rocks. But fails to mention that the tools to find small rocks also help us identify big rocks with more clarity.

The size of the sample depends on how rare of an event that is being studied. For something that happens 80% of the time, 300 is a very reasonable number. Something that happens 1/1000 or 999/1000 would need a much larger sample size.

Yes, I agree. But that’s to the same point. The p-value is still a measure of whether your sample size is large enough to capture a small probability event in the context of assessing the null hypothesis