News for hackers. Stuff that matters.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by PaulKeeble on Monday July 10, 2023 @09:47AM (#36666377)
Almost no findings in psychology survive 20 years, almost all of them are determined to be fraudulent manipulation of data and findings or at the least fail to be replicated. Psychology is not producing scientific findings as a field, it's completely ignorable and responsible for a large chunk of the science replication crisis.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by UncleMeat on Monday July 10, 2023 @10:27AM (#36666957)
Psychology is also one of the only fields funding replication.

My experience in CS is that the replicability of experimental results is embarrassingly bad, but this isn't making headlines in the same way so people don't consider it to be a problem.

This is data fraud in psych. Data fraud has also happened in plenty of other fields. When it occurs in these fields it is seen as a one-off. When it occurs in psych, it is because the entire field is useless garbage. That's not the lesson to take away here.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by stcroixx on Monday July 10, 2023 @10:34AM (#36667064)
Well, CS doesn't involve pharmaceuticals so the harm caused by fraud isn't as serious.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by UncleMeat on Monday July 10, 2023 @10:41AM (#36667162)
We could maybe make the conversation about the social penetration of ideas coming out of various disciplines and how invalid research can harm policy or medical practice, but that'd be a completely different discussion than "psychology is utter garbage as a field."
Re:Harvard ethics prof fabricated multiple behavioral scienc... by manicennui on Monday July 10, 2023 @10:41AM (#36667164)
Because no one writes software that can lead to death.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by friendly_wizard on Monday July 10, 2023 @11:23AM (#36667766)
Tell that to air traffic controllers
Re:Harvard ethics prof fabricated multiple behavioral scienc... by philwelch on Monday July 10, 2023 @12:02PM (#36668307)
Therac-25.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by ajuc on Monday July 10, 2023 @10:51AM (#36667329)
You can easily kill people with software. It has happened.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by triceratops on Monday July 10, 2023 @11:51AM (#36668164)
Isn't that psychiatry? Psychologists aren't necessarily licensed medical doctors who can prescribe.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by justrealist on Monday July 10, 2023 @10:44AM (#36667208)
> My experience in CS is that the replicability of experimental results is embarrassingly bad

I think that's true for low-tier/low-impact CS papers, but the difference is that literally nobody cares about those papers. The high-tier stuff is easily verifiable (like Tensorflow or whatever) and nobody is writing articles about 5% improvements against a benchmark in some obscure niche scheduling and planning domain.

Outside academic CS people are more scientific about experimental results because it has concrete implications on revenue or spend... but they aren't getting the results from conference papers.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by wongarsu on Monday July 10, 2023 @11:18AM (#36667693)
The ML part of CS is in a sense funding replication: there are a decent number of papers whose only premise is "we compare 5 recent papers on this benchmark (and fill a couple pages with discussion)" or "we made a new benchmark, here's how commonly cited papers compare (plus a couple pages of discussion)"

Outside the high-profile cases it seems accepted norm that papers perform far worse when scored against somebody else's benchmark. The real measure of quality is how big the gap is.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by UncleMeat on Monday July 10, 2023 @10:51AM (#36667322)
I'm talking about papers in top PL and Security venues.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by mrangle on Monday July 10, 2023 @11:15AM (#36667655)
Other fields have a fairly firm scientific foundation. Whereas it is routinely pointed out by psychologists and psychiatrists that psych has no such thing. The field struggles to authoritatively explain the first thing about human thinking.

Adding to complication in the psych field is the widely observed phenomenon that it draws students with psych problems. Like those who would be willing to fabricate data, for example.

But hey, at least it isn't sociology.

There are resilient exceptions in academia in general, but soft science fields have led the way in the decay of academic standards.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by mike_hearn on Monday July 10, 2023 @03:16PM (#36671513)
Psychology really isn't the worst field out there. They do at least talk about their problems and claim to care about them, which is better than most fields. They actually do try to replicate each other's results sometimes and publish it when they fail. I used to believe psychology must be uniquely bad, these days I think it's just uniquely honest (!).

There are dozens of well known fields that are built on epistemological quicksand yet which point-blank refuse to admit to or talk about their problems. Because they have a culture of deny deny deny, even when the evidence is overwhelming, they actually get less attention because why bother doing a nice writeup of fraud if you know the result will just be stonewalling? It's better to focus on fields where there might be some actual response, a chance of improvement, no matter how minor.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by joshe on Monday July 10, 2023 @12:12PM (#36668424)
No. Psychology is the worst. It is encouraging that an insurgent group is taking replication on but it feels like only about 20% of the field.

Some results do replicate. Big Five personality traits, general classification of mental illness, mainstream IQ results, and human perception and performance. All psychology is not false and there are interesting things to learn from the field.

But the severity and frequency of fraud and non replication is worst in psychology. So much so that half of what you learned in Psych 101 ten years ago does not replicate.

The big challenge for the field is that humans know a lot of psychology. We can track our own thoughts and we are constantly interacting with people and trying to understand their psychology. A biologist who studies ants intensely for 5 years is maybe only one of 100 people have ever watched them that carefully. They'll find lots of new stuff.

Psychology doesn't have powerful techniques than that for determining new truths about humans. It's still mostly give a questionnaire or put people in weird situations.

So there are a lot of researchers hunting around for original ideas that are undiscoverable with our current tech. Some of them are bound to give into the temptation to just make up an interesting result with manufactured data. With p hacking they might not even be committing fraud, they are desperate for a positive result, when the get one they stop looking and publish.

So I would still guess in 2023 that about 80% of "new" discoveries published in journals won't replicate. That rises to 95% with journal publications that are published as university pr and reported in the media.

For coverage of what to remember to unlearn Rolf Degen is great: https://twitter.com/DegenRolf

Popular examples that do not replicate (mostly from Rolf):

Repressed childhood memories Social media harms Search bubbles and echo chambers Women prefer masculine men when fertile Priming (you see a fight in the hallway and then don't cooperate later) Power posing (puff out your chest to feel more confident) Watching eyes make people more honest (in Kahneman and Malcolm Gladwell)

Notice that a lot of these are interesting, and you want them to be true. That's not enough to make it so.

As a bonus, an interesting interview with Daniel Kahneman: https://www.edge.org/adversarial-collaboration-daniel-kahnem...

Re:Harvard ethics prof fabricated multiple behavioral scienc... by klysm on Monday July 10, 2023 @09:53AM (#36666459)
An epistemological crisis rooted in a shitty incentive system and powered by bad statistics
Re:Harvard ethics prof fabricated multiple behavioral scienc... by pyrale on Monday July 10, 2023 @10:21AM (#36666873)
There are differences, however, between research using poor methodology that end up being worthless because it can't be replicated, and research where some data points were deleted and others were created in order to show a better result.

The work here isn't fraudulent because of a mistake or because the author has shaky statistics knowledge, but because it was fabricated.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by vgalin on Monday July 10, 2023 @10:05AM (#36666646)
If this is an established fact, where could one learn more about this?
Re:Harvard ethics prof fabricated multiple behavioral scienc... by taeric on Monday July 10, 2023 @10:10AM (#36666705)
It is, at least, fairly well accepted as a thing. Name a "fact" that has been accepted as a learning, dive into it, find that it hasn't held up. :(

Some are purely on the popular science side of failure. The 10k of practice was basically thrown out, but the original research still seems good. They never claimed "if you do 10k of work, you will be good."

The "marshmallow test," though, seems completely tossed? Maybe there was something there?

Anchoring and other items? Not sure how well those have survived. :(

https://en.wikipedia.org/wiki/Replication_crisis looks to be a good article, though I haven't finished reading it.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by scythe on Monday July 10, 2023 @10:33AM (#36667052)
Some of Kahneman and Tversky's work was thrown out, but the majority of it replicates:

https://www.nature.com/articles/s41562-020-0886-x

Re:Harvard ethics prof fabricated multiple behavioral scienc... by PaulKeeble on Monday July 10, 2023 @11:38AM (#36667995)
The Dunning Kruger data is also just a statistical manipulation.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by mike_hearn on Monday July 10, 2023 @03:20PM (#36671558)
Worse: the original Dunning Kruger experimental design doesn't even make sense.

https://news.ycombinator.com/item?id=31119836

Re:Harvard ethics prof fabricated multiple behavioral scienc... by endominus on Monday July 10, 2023 @10:57AM (#36667396)
Per https://www.gleech.org/psych:

- Stanford Prison Experiment; Not an experiment, abuse was scripted, experimenter constantly intervened, reactions of participants were faked, and there was not even a scientific hypothesis they were testing. Definitely read the paper[0] debunking it.

- Milgram Experiment: (the one where people were ordered to shock actors) No good evidence for it. Researchers did not follow script, implausible levels of agreement between different experiments. Killer line is, “only half of the people who undertook the experiment fully believed it was real and of those, 66% disobeyed the experimenter.”

- Robber's Cave: (the one where two groups of kids immediately formed tribal hatred between one another) The conflict was orchestrated by experimenters and the experiment was actually repeated because the first time the kids absolutely refused to turn on one another. More information at [1].

- At best, weak evidence for implicit bias testing and stereotype threat.

- Weak evidence of "facial feedback" (smiling causes a good mood and frowning causes a bad mood)

- Good evidence against "ego depletion" (the idea that willpower is limited in a muscle-like fashion)

- Mixed evidence for Dunning-Kruger effect

- Questionable evidence for "hungry judge" effect (the idea that judicial sentences are massively more merciful in the morning and after a lunch recess due to "ego depletion" - this is also thoroughly debunked here [2])

- The 10,000 hours of practice leading to expertise idea has been disowned by its proponents

- No good evidence that tailoring teaching to students’ preferred learning styles has any effect on objective measures of attainment.

- No good evidence that brains contain one mind per hemisphere. i.e. the left-brain, right-brain split that people talk about, especially after the link between the hemispheres is severed.

- No good evidence for left/right hemisphere dominance correlating with personality differences.

Per https://danluu.com/dunning-kruger/:

- Most people talking about Dunning-Kruger have no idea what it actually means. The actual purported bias is much weaker than people claim - basically, that everyone either overestimates their ability, but that estimation is still positively correlated with actual ability, or estimated ability has basically no correlation with actual ability and everyone is just guessing.

- Increasing your wealth does in fact make you happier at a predictable rate. There is no "plateau" of wealth or income - what appears to be a plateau is misleading displays of data. In effect, increasing income by a proportional rate will increase reported happiness by a fixed rate. For example, say you make $10, and your happiness is 50. Then, your income increases to $20 and your happiness increases to 60. Then, doubling your happiness is necessary to increase your happiness by 10. It's a logarithmic function; plotted on a standard axis, it looks like a plateau, but plotted on a logarithmic scale and it's a constantly increasing line.

- Hedonic Adaptation (aka the hedonic treadmill) is a myth. Bad life events (divorce, disability, death of a loved one) all have negative long-term effects on happiness. Vice versa for positive events.

[0]: https://www.gwern.net/docs/psychology/2019-letexier.pdf

[1]: https://www.theguardian.com/science/2018/apr/16/a-real-life-...

[2]: http://daniellakens.blogspot.com/2017/07/impossibly-hungry-j...

Re:Harvard ethics prof fabricated multiple behavioral scienc... by AzzieElbab on Monday July 10, 2023 @11:42AM (#36668042)
Not to mention the grievance studies affair and the Sokal hoax. Next level

https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Sokal_affair

Re:Harvard ethics prof fabricated multiple behavioral scienc... by booleandilemma on Monday July 10, 2023 @10:04AM (#36666631)
It sure does give you some cool factoids to bring up at parties though. Did you hear the one about the prison experiment?
Re:Harvard ethics prof fabricated multiple behavioral scienc... by blululu on Monday July 10, 2023 @10:24AM (#36666918)
The interpretation is never so simple. I'm not sure what that guy at Stanford was trying to prove but his research conclusively proved that tenured psych professors at leading universities can be wildly unethical. This discovery helped support the creation of institutional review boards across the country - one of the first major restrictions on academic freedom in the history of science.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by sandworm101 on Monday July 10, 2023 @10:32AM (#36667032)
Or they proved that if you setup some people to work/live in a fake prison, they end up acting basically like all the other people who already work/live in realworld prisons. It's like having people sit in a MacDonald's for ten hours each day, then writing a paper about how subjects ate more fast food during the trial period. Breaking news: Starting a new job in Hawaii increases the likelihood that you will take up surfing.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by posterboy on Monday July 10, 2023 @10:19AM (#36666849)
Or Psychology is difficult and having passed the the bar is not a quality controlled guarantee, same as for example nautical engineering isn't. That's not enough reason to say that boats don't float.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by NoMoreNicksLeft on Monday July 10, 2023 @10:24AM (#36666914)
What is psychology used for, and whatever those things are... are they still floating (so to speak)?

Some of it is used in policy. So when some convicted murderer is out on furlough and kills again, am I wrong to wonder if that was some of the top-notch applied psychology at work?

Re:Harvard ethics prof fabricated multiple behavioral scienc... by posterboy on Monday July 10, 2023 @10:56AM (#36667386)
That's shifting the goal post and it suffers from survivor bias. The individual decision is met by few people. The policy is instated by very different people and informed by many factors outside the control of psychology, where false positives don't invalidate the whole system if the alternative is a slippery slope with draconic punishments – a paradox because the most severe punishment could amount to manslaughter if not murder.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by hx8 on Monday July 10, 2023 @10:54AM (#36667354)
This is such an exaggerated claim.

* Plenty of studies inherently cannot survive 20 years. Send out a survey and ask people "on a scale of 1 to 10 how happy are you". These results will not match the results in 21 years simply because the population you are studying is different. This example is simplified, but explains why many results change over time.

* Plenty of studies have survived 20 years, and those that study psychology are really clear about which knowledge is foundational and which is on more questionable ground. The Big Five personalty traits is about 40 years old and has been found to be consistent across cultures. A ton of psychological research around psychological responses has survived the test of time.

* Psychologist often relay on more questionable data than other scientists because it's much cheaper and easier for psychologist to collect questionable data than other sciences. This allows for a wider exploration and makes research much more accessible. Anyone reading a paper with this type of data will be naturally skeptical. I am specifically referring to data collected from self-reported questionnaires, often collecting a sample that generalizes poorly over interesting populations.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by tgv on Monday July 10, 2023 @11:02AM (#36667477)
As I've written more than once here: it's are harder to experiment upon than physics. Society/individual behavior is also way more complex than e.g. rheology or organic chemistry, and doesn't submit itself to lab research.

But that's not the only problem. The problem is that people try it anyway. They have some hypothesis within a theoretical framework which is embedded in other frameworks, none of which is proven. How could they be? But, they set up an experiment anyway, and (usually after a couple of attempts) they find something that can reject H0. Then they publish an article stating that their theoretical framework is a fact.

There is so much wrong in this process, even ignoring the manipulation of the stimuli and conditions, and the statistical procedures, yet the theory has a good chance to make school and become the subject of research in dozens of psych departments, each contributing articles to the literature about it. After a while, an essentially flawed theory enters the handbooks.

Then, if you want to make a name for yourself as a researcher, and you need publications, you can simply attack older theories. They'll crumble like cookies, you get your publications, and the cycle restarts.

And as critical as I am of psychology, other social sciences have even lower empirical standards. Educational sciences, sociology, linguistics all have very little to show for a century of research. And that's ignoring disciplines like political sciences or history.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by BeetleB on Monday July 10, 2023 @11:09AM (#36667563)
> and responsible for a large chunk of the science replication crisis.

Former semiconductor researcher here. It is quite common for researchers not to believe published papers - even in prestigious journals from well known researchers. No one bothers replicating, and they don't put enough information in the paper to replicate (competition - they don't want others to get the secret sauce).

This was true for both computational and experimental papers.

Oh, and I did know one person personally who falsified data (and was caught). He just transferred to another prestigious school and got his PhD there instead.

It was quite demoralizing and helped me decide not to pursue academia.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by rcxdude on Tuesday July 11, 2023 @10:02AM (#36681118)
I think a lot of the communication around science publishing has overemphasized the power of peer review. More people should realise it's the table stakes for science, not the conclusion.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by somsak2 on Tuesday July 11, 2023 @02:50AM (#36677277)
Any sources for these claims?