"After the tragedies caused by the use of thalidomide in pregnant women, the FDA issued “General Considerations for the Clinical Evaluation of Drugs” in 1977. This guidance document stated that women of child-bearing potential should be excluded from Phase 1 and early Phase 2 research, except if these studies were being conducted to test a drug for a life-threatening illness. If a drug appeared to have a favorable risk-benefit assessment, women could then be included in later Phase 2 and Phase 3 trials if animal teratogenicity and fertility studies were finished...In 1993, FDA reversed the 1977 guidance with another guidance document entitled Guidelines for the Study and Evaluation of Gender Differences in the Clinical Evaluation of Drugs."

Also:

"In a study that evaluated the inclusion and analysis of sex in the results of federally-funded randomized clinical trials in nine major medical journals in 2009, researchers found most studies that were not sex-specific had an average enrollment of 37% women."

From "Women’s involvement in clinical trials: historical perspective and future implications" - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4800017/

They could be attractive to the desperately impoverished single adults, who are mostly men.

Stats on poverty by sex and age in the United States: https://www.statista.com/statistics/233154/us-poverty-rate-b...

Worldwide, the gap is even wider: https://www.washingtonpost.com/news/worldviews/wp/2018/02/14...

I should have been clearer than simply stating _desperately_ poor; I intended to convey that they are in a state of poverty where there are few other options available.

0: https://en.wikipedia.org/wiki/Homelessness_in_the_United_Sta...

1: https://www.forbes.com/sites/eriksherman/2018/12/29/homeless...

I wonder why that is, I would've thought poverty and homelessness goes hand in hand.

Maybe it's more to do with mental health? Of all the homeless people I knew, a sizeable portion of them suffered from serious mental illnesses.

I don't know, I just found it interesting.

It's a really horrible place to be.

Men are the disposable sex.

And the world stats aren't relevant as we're talking about psych studies that mostly take place in the US.

social psychology experiments use whoever they can get, often for class credit only

There are many analogs of that in Computer science too. For example in computer vision, CIFAR-10 is a de facto standard for measuring performance. A set of 60k 32x32 images. Good results on that data set doesn't necessarily translate well into real-world performance. But what can you do? Gathering huge data sets and having humans annotate them is incredibly expensive.

Another ml example is music recognition. There are several state-of-the-art methods for detecting notes in music. So one Chinese researcher tried to apply them for Jingju music https://www.youtube.com/watch?v=NJsTl342RhI. The results were... less than stellar.

> subset of 18 to 21 year old American women.

This is perfect!

I had a similar epiphany about linguistics, much of which appears to be the study of example sentences that linguists come up with.

Since you could invariably tell when something was an example sentence, this was obviously a different language from what people actually used. (Of course there are linguists who go out and study real language-use, but some actively dismiss this as mostly irrelevant "performance")

" [...] This suggests that even under the (likely unwarranted) assumption that the discrepant results are all false positives that have found their way into the syntactic literature due to the shortcomings of traditional methods, the minimum replication rate of these 469 data points is 98%."

98% replicability!! Compare that with psychology.

The first problem is that asking people how they speak or what is acceptable does not accurately represent how they actually speak when not observed/asked:

https://en.wikipedia.org/wiki/Observer%27s_paradox

The second problem is that, even if the first isn't an issue, the most this can demonstrate is that linguistic example sentences are a subset of actual language.

I'm not sure about the amount of research but the amount of psychology students here in the Netherlands is huge. And it seems to function in much the same way. Often first-years get some form of credit for mandatory participation in experiments for higher-year students and sometimes regular faculty members. I'm also aware of some researchers going out of their way to get a different sample from the population but in the end it's a lot easier to get participants when you make them.

I don't know any easy way out here. Just because "folk psychology" can say formal psychology is too simple doesn't mean folk psychology and anecdote are more useful.

Wait, does participating in a study as a subject count for credit, or does working on a study count?

Aren't studies explicit about the demographics of their participants? And don't studies that make generalized claims usually control for things like gender and age?

Some overlap does exist, but it’s significantly less than suggested by the requirement.

It was fun, but it was seriously disillusioning to see how absurd most psychology studies are. Two researchers talk to me for 5 minutes about women performing better than men on certain forms of math tests. They then sit me in a room alone for about 10 minutes presumably to try to make me nervous. Finally they give me a test, overtly handing me it on a pink sheet of paper instead of a white one. Then after the test they continually tried to get me to agree that I felt like I probably didn't do so well on the test. I refused - since I was certain I nailed it (it was a cookie cutter pre-sat math type test). They told me I'd receive my results by mail within 2 months. I never did. E-mails asking for such were ignored. The research got published and was of course some trite gender bias explains all difference in math paper.

At the time I just found the whole thing quite absurd. In hindsight, I think it was even worse. That study was prepped and ready for p-hacking. They were measuring and possibly varying a large number of variables there. There was the paper color, the isolation, the 'chat(s)', different sections of the test, were men/women over/under estimating their performance, etc. And they were actively trying to push people into agreeing with what they wanted to find. You're going to be able to find some sort of statistically significant correlation there, even if none actually exists.

> Wait, does participating in a study as a subject count for credit, or does working on a study count?

There was this course called "measurements and diagnostics" and for that you had to pass the theory/tutorials of the course and also 10 hours of being a participant in research. If you didn't do the 10 hours of participation, you could do them at any moment you want and if you completed it, you immediately passed the course.

My university has about 400 psychology students, so that is 4000 hours worth of people participating per year. And I'm pretty sure that more universities are doing this. In The Netherlands alone I estimate there to be about 40000 psychology students per year and probably about 5 hours of compulsory 'participation' in experimental studies. So a tiny country like The Netherlands is skewing this particular bias about 20000 participant hours per year.

Participating in experimental studies is quite fun (pro tip: go for the fMRI and EEG studies ASAP! ;-) ), but I didn't appreciate how I was forced to do it. I didn't learn too much from those experiments and I felt used. It furthermore is a way to be lazy regarding participant recruitment. I'm pretty sure that some startup could disrupt this particular problem. If you have ideas about it, email me, I'm willing to brainstorm and help you to look for ways to solve this problem.

> Aren't studies explicit about the demographics of their participants? And don't studies that make generalized claims usually control for things like gender and age?

Studies are explicit about the demographics, which is why the WEIRD article could be written. And yes they do control for things like gender and age and in general they are quite careful making those claims. The issue is though as a research field psychology isn't the study of the human mind and human behavior but of WEIRD people their mind and WEIRD people their behavior.

Like a lot of societal issues that are in popular media today, this is a systemic issue. Though I could imagine psychology researchers being like: "Hmm, I could get a representative group or I could just get some psychology students and churn my next paper out twice as fast!"

Here's another interesting phenomenon: a certain personality type (of which I may or may not count my self among) will deliberately supply false data which attempts to invalidate what we perceive as your study goal when forced to participate against our wills in this way. Its a small rebellion against an academic system that doesn't respect us or intellectual integrity, but is focused on pumping out shoddy research on unwilling participants.

Turn academics into a game and thats how it'll be treated.

I also have the choice of not working or wiring some other job.

I would say it's a responsibility not supporting forced on people against their will. Two very different things.

Voluntary informed consent generally requires that participation is completely voluntary, that any rewards for participating don't cause undue influence on the participant's decision to participate and that there will be no consequences or loss of benefits for not participating.

I would argue that to reach this standard the requirement to participate in studies to complete your degree would need to be known before enrollment (likely the case) and the actual study you'll be participating in needs to be identified (almost definitely not the case). Anything less and students will find themselves financially and personally committed to a course of education without truly knowing what they'll be required to do to pass.

If you're interested in the topic, take a look at this neat little booklet: https://oprs.usc.edu/files/2017/04/Informed-Consent-Booklet-...

I don't see much ability to innovate here other than Mechanical Turking out the work to people you pay to participate, but if you come up with something interesting I'd love to hear it…

You'd be surprised how common that is.

I'm use to the alternative being a research paper that can easily take 20 times longer to complete than being a test subject.

I'd add in Beagles, Capuchins, Zebrafish, and Mongolian Gerbils there too.