News for hackers. Stuff that matters.
Re:About 40% of economics experiments fail replication surve... by rossdavidh on Sunday September 16, 2018 @10:45PM (#18002786)
So, their take is that 60% replication is good news. Yikes.
Re:About 40% of economics experiments fail replication surve... by air7 on Monday September 17, 2018 @04:45AM (#18004172)
With the current Publish or Perish zeitgeist in science this number can be seen as high.

Also, even if assuming legitimate significant results, it's hard to control for every possible confounding variable: Maybe part of the effect is due to some sociological parameters of of the test group, maybe it's the heat/cold, maybe it only happens on Tuesdays. Even the same stimulus can change its meaning over time (like the Williams video). Humans make for finicky test subjects.

IMO a single scientific paper shouldn't be considered much more than a small indicator that an effect might exist and it will require further work. I think popular media does Science a disservice by ignoring this and headlining single papers as "fact".

So given that, 60% is not that bad...

Re:About 40% of economics experiments fail replication surve... by rossdavidh on Monday September 17, 2018 @11:03AM (#18006551)
True, but also not that good. All of the heat/cold/only on Tuesdays stuff should have been baked into the original experiment, really. Not that they should be able to take into account everything, but 60% replication does not indicate a very high standard. I mean it's not much better than if you just took and educated guess, with no data, since you could probably do better than 50/50. The whole point of the study is that it is supposed to be better than an educated guess.
Re:About 40% of economics experiments fail replication surve... by hannob on Monday September 17, 2018 @04:44AM (#18004164)
I hate to break it to you, but that's the given standard of science.

There's pretty good evidence that a mixture of publication bias and p-hacking very likely leads to the majority of studies being false: https://journals.plos.org/plosmedicine/article?id=10.1371/jo...

Given that in most fields publication bias and p-hacking is standard and basically written into the way science is published (i.e. you only publish your results after you've done your study, so noone knows what you intended to study when you started) it's normal to assume that most things published are simply wrong.

Re:About 40% of economics experiments fail replication surve... by lsh123 on Sunday September 16, 2018 @11:02PM (#18002855)
Better than 50-50 I guess
Re:About 40% of economics experiments fail replication surve... by thaumasiotes on Sunday September 16, 2018 @11:27PM (#18002961)
OK, but why 50-50? There's no reason to expect that a priori. It's also better then 55-45 and 20-80, but none of those is good.
Re:About 40% of economics experiments fail replication surve... by forgingahead on Sunday September 16, 2018 @11:36PM (#18003012)
I think he was being sarcastic and highlighting the absurdity of the situation.
Re:About 40% of economics experiments fail replication surve... by justtopost on Sunday September 16, 2018 @11:35PM (#18003003)
I think he was implying that better than 50% implies its at least not mostly bullshit, or that we could perhaps draw usful inferrences if we are right at least most of the time. But agreed, none are particularly enthusing.