Re:Marshmallow Test Fails to Replicate
by kjkjadksj on Saturday August 03, 2024 @01:37PM
(#41147976)
This is one of those rules of thumb that don’t make any sense to me as someone who works with data in the field. You conduct the same exact experiment with the same conditions and same data and get the same result. But whether you speculate on a hypothesis before or after suddenly changes the significance threshold with no actual change in the underlying data or method? Did you cast some ancient spell when you came up with the hypothesis or something?
[ Parent
| on HN ]
Re:Marshmallow Test Fails to Replicate
by marcosdumay on Saturday August 03, 2024 @03:12PM
(#41148565)
> with no actual change in the underlying data or method
Proving a known hypothesis or deciding what you want to know after the fact are completely different methods.
> Did you cast some ancient spell when you came up with the hypothesis or something?
"I'll know it when I see it" is an incredibly vague way of doing science that requires extra rigor somewhere else to compensate.
Or, in a more rigorous explanation, testing for multiple hypothesis is subject to that kind of failure:
So you need more data confirming your theory.
But if you state your single hypothesis beforehand, you are at the situation of the top square, with a high-confidence result.