BTW, I strongly disagree the quote you have about statistically insignificant data. It cannot be used to say there is no effect, or anything about the direction or magnitude of the effect. I'm not sure what a statement like """somewhere in their schooling they were falsely taught that results need to be “statistically significant” to be “real.”""" even means, significance is a mathematical/probabilistic formalism that has nothing to do with underlying reality, just with the event system you're working on.
BTW I set my p threshold at 1e-6 for a single test, 0.05 is just crazy. On the other hand, I've worked with systems where nobody cared about the significance, just that the weights that got pushed to production made more money than the last set of weights.
Deep within the linked piece there is a small section entitled 'On “statistical significance”'. It says this:
Now, this result does not happen to be “statistically significant.”
People often use arguments like “but it’s not statistically significant” to try to discount results. I even hear people with PhDs or MDs say this, because somewhere in their schooling they were falsely taught that results need to be “statistically significant” to be “real.”
But even if a result is not statistically significant, absent better data, it still remains the best available estimate for the effect in question. This is just common sense. “Statistically insignificant” does not mean that there is no effect. More on statistical significance here: https://www.nature.com/articles/d41586-019-00857-9.
As a non-PhD who has (relatively) thought a lot about statistics, and who has previously linked others to that Nature article, this is the sort of balanced statement that makes me think she is not clueless. How about you?