This was linked from the article, BTW - and the responses they also linked to are dead on:
http://mchankins.wordpress.com/2014/07/12/do-not-replicate
http://vserver1.cscs.lsa.umich.edu/~crshalizi/weblog/1102.ht...
Which one is it?
You can not tell.
There is an old saying: "A man with a watch knows what time it is. A man with two watches is never sure." - http://en.wikipedia.org/wiki/Segal%27s_law
Traditionally, replication hasn't been done in psychology because the experiments have to be set up in really clever complicated ways in order to tease out effects. Even the most dedicated replicator can't ever fully replicate a psychology experiment. You'd have to ship the actors used as confederates across the country to do that and of course nobody does that. It's foolish to think any psych study has ever been replicated.
The way psychologists deal with this is the concept of converging operations. This has to do more with being clever and considering implications of a thing you're trying to study with respect to other theories you have much more evidence for and in different situations. If you think stereotype threat is a thing, for example, it would make sense for stereotype boost to exist if the stereotype is positive. If a priming effect speeds up the recognition of "doctor" after a subject has been shown "nurse", maybe recognition for "warmth" will be slower after a subject has been shown "frozen". If people only encode based on schema variables, they'll be unable to remember information unrelated to a schema even if they're provided with congruent encoding cues.
Converging operations is just studying the same concept from different angles and seeing if you keep seeing consistent results. If the results aren't consistent the hypothesis you're investigating will, in fact, die. Psychology is an advancing science as of right now. If people were just publishing random noise there'd be no reason for concepts to fall out of favor.
I think people on hacker news tend to wildly underestimate the degree of skill involved in experimental psychology. So much of statistics, even, has come out of psychology because of the need to refine and clarify what effect you're seeing.
Alternatively, if you think because of this existing academic psychology is lacking, you're in a fantastic place to be. Do literally anything to get some seed money, just working in a standard SDE job would be fine, and then run some small experiments (and replicate them, of course) on learning or addiction. Then write the next Farmville or build the next Facebook. If there's really that much low-hanging fruit, go out and pluck it. Put up or shut up.
Now wait a second. When a psychology study comes out, it claims "This experiment shows stereotype thread reduces performance of Honduran men on a Math test." Such studies rarely claim "This experiment shows stereotype threat reduces the performance of Honduran men on a Math test when Jill the skinny experimenter repeats the code words."
If using different experimenters yields a different result, it means the effect being described is probably not as robust as the experimenters claim. The cause might be Jill's shifty eyes rather than stereotype threat.
That's a replication failure, in the sense that it shows the claimed effect is far weaker (or causally different) than the original study claimed.
If you have more than one researcher administering a test in a stereotype threat experiment, that'll reduce the likelihood of that being the confound. If other studies (not replications, but their own experiments, with their own, different approaches to study the problem) agree with the first experiment, the effect is likely real.
I would also say that you're viewing the entire system far too antagonistically. Nobody goes into academic psychology to get rich off of it.
Your reasoning here is as fallacious as the idea that unrepeatable experiments are good science.
Your psychological hypotheses on addictive mechanisms can be entirely false and yet your game can still be addictive. Maybe the Skinner Box-like environment you create has a particular applicable demographic outside which it is largely impotent; but failure to repeat the experiment isn't seen as contradictory to the hypothesis and so you fail to address different markets with the specific sort of addictive elements that suit them best. You'd be leaving a heap of money on the table [for shame /s].
The ancient Greek idea that gravity worked by the attraction of Earth-kind substances to the Earth itself was falsified and yet things still fell down.
Would you say that a single superstitious action (finger crossing, say) coinciding with the desired outcome proves that finger crossing works? Do you think that those who ascribe to such a progression of logic as demonstrating sound scientific thinking, are being scientists?
IMO observations and experiments that can't be repeated are of the utmost importance in furthering knowledge. Truthfully reported anecdotes are valid data. But unrepeatable events have limited scientific value in and of themselves, they can not show statistical significance nor demonstrate that a hypothesis should be held to be true as an objectively valid scientific theory.
I did indicate that I'm happy to accept other methodologies as beneficial and am certainly not wedded to positivistic approaches in all fields of human understanding. Are you perhaps pushing the meaning of "repeatable" in the current context beyond the bounds of "supporting the same breadth of conclusion with equal confidence".
It is however not scientific to rely on unrepeatable experimental results. If an experiment is unrepeatable then the conclusions are proved false. Note that the question at hand is not whether the initial experiment is valid or useful. Science after all is axiomatic and the world, generally, non-deterministic so far as one can tell [though I know there are many dissenters on this point].
Rather than arguing from abstract implications about my assumed knowledge why don't you address the question at hand?
This is why this string of multiple replication failures and fraud scandals have turned the spotlight to these practices from within the discipline, with reasonable and high-profile people openly questioning their validity and pushing for a whole "replication movement" [1-4]. So it's not just HN commenters, you know.
[1] http://www.nature.com/news/replication-studies-bad-copy-1.10...
[2] http://chronicle.com/article/Power-of-Suggestion/136907/
[3] http://www.psycontent.com/content/311q281518161139/fulltext....
[4] http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinf...
[1] http://neurochambers.blogspot.com/2012/03/you-cant-replicate...
The thing is amazing to read, in the "someone who thinks he's a scientist actually wrote this?" sense, like this gem:
Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way.