Thanks for the response. Do you know of any you good blog posts or articles that dive into this a bit more? It looks very interesting.
This is the best description of the main problem (testing your own vs some default hypothesis) I have seen:
Paul E. Meehl, "Theory-Testing in Psychology and Physics: A Methodological Paradox," Philosophy of Science 34, no. 2 (Jun., 1967): 103-115. https://doi.org/10.1086/288135
Free download here: www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf
Andrew Gelman (andrewgelman.com) has a great blog that often touches on this issue.
Thanks a bunch for sharing, I appreciate it. I'll add these resources to my reading list. I may also pass it along to my brother who getting a graduate degree in psych :)
EDIT:
I guess I should say that the concept of testing a "null model" without interpreting the fit relative to other models is wrong to begin with. You need to use Bayes' rule and determine:
Lots of stuff wrong with what has been standard stats for the last 70 years, it literally amounts to stringing together a bunch of fallacies and makes no sense at all.