News for hackers. Stuff that matters.
Re:Could It Be? Spooky Experiments That 'See' The Future by bigfudge on Tuesday January 04, 2011 @07:55PM (#2069302)
Ok, then let's funding psychology like we fund physics. I would love to run 1000+ patient studies to test psychotherapies, and in fact we'd be able to answer some really interesting questions if we did, but there is currently no way of doing this.
Re:Could It Be? Spooky Experiments That 'See' The Future by Eliezer on Tuesday January 04, 2011 @11:17PM (#2069862)
I repeat, you do not need 1000 times as many subjects to get results that are 1000 times as significant! If 40 subjects gets you results with p < 0.05, then 100 subjects should get you results with p < 0.001. Doing half as many experiments and having nearly all the published results being real effects, instead of most of them failing to replicate when tested, sounds like a great tradeoff to me.

And I suspect the ultimate reason it's not done this way... is that scientists in certain fields would publish a lot fewer papers, not slightly fewer but a lot fewer, if all the effects they were studying had to be real.

Re:Could It Be? Spooky Experiments That 'See' The Future by saurik on Tuesday January 04, 2011 @11:36PM (#2069919)
"1000+", not "1000x". Also, I'm assuming bigfudge was talking about p < 0.0001, given the comparison made to physicists.
Re:Could It Be? Spooky Experiments That 'See' The Future by bigfudge on Wednesday January 05, 2011 @09:35AM (#2071050)
Yes - thanks. The current norm for a 'suitably powered' trial of a psychotherapy is about 300. We've just got a trial funded for that number (admittedly in a challenging patient population) which will cost about £2.5m in research and treatment costs. We would love to run 1000 patients and start looking at therapist-client interactions, individual differences in treatment suitability but that's out of the question.
Re:Could It Be? Spooky Experiments That 'See' The Future by tokenadult on Tuesday January 04, 2011 @10:03PM (#2069659)
Let's fund psychology like that when the psychologists define their hypotheses before the experiment begins as well as physicists do.
Re:Could It Be? Spooky Experiments That 'See' The Future by bigfudge on Wednesday January 05, 2011 @09:39AM (#2071062)
That's a cheap shot. Our trial will publish a detailed protocol and analysis plan, as do most large, publicly funded trials. Small-scale experimental work is a different matter. I personally agree that all experiments which could end up in peer reviewed journals should be registered before participants are recruited.

This would be simple to do by submitting ethics applications and an analysis plan to a trusted third party which would only release them once the author is in a position to publish, or at a pre-agreed cutoff (perhaps 2 years), whichever is the shorter (to avoid scooping). Perhaps I should set something up...