News for hackers. Stuff that matters.
Re:Ketamine lifts rodents' mood only if administered by male... by gwern on Tuesday November 21, 2017 @10:30PM (#15754016)
I never did manage to trace Feynman's story back to a source, but nevertheless, also something Derek Lowe repeatedly covers for biology/chemistry on "In the Pipeline": a lot of the 'cell lines' or 'reagents' used in lab work turn out to just not be what they're supposed to be, either because they never were or they degraded into something else in storage or something. So to give a recent paper he blogged, there's something like 30k papers using "immune system cells" which are actually just HeLas; presumably the cell lines in question got contaminated or swapped somewhere along the way, but that raises the question of what any of those papers mean...

And of course there are similar issues with mice. You have all the different strains which are genetically different, but they'll also be genetically changing over time through drift & selection & mutations, so your Wistar-XXX may not be the Wistar-XXX from 50 years ago. Or mice chow - sure, the label may say it's the same but when was the last time you checked? (Think about that paper about how all sorts of lab animals, supposedly fed constant diets and under constant controlled environments, keep getting fatter over the past century. Something is changing; what? Genetics? Food composition? Micronutrients? Lab infections? Microbiomes? Measurement tools themselves?)

Re:Ketamine lifts rodents' mood only if administered by male... by jaggederest on Wednesday November 22, 2017 @02:48AM (#15755068)
I think it would be very interesting to have labs run detailed A-A studies to investigate these kind of things.

"This kind of mouse we raised under these conditions, and we changed nothing, but they varied in these ways over time"

I often wonder whether the variables we check against are actually as independent and subject to the central limit theorem as we assume.

Re:Ketamine lifts rodents' mood only if administered by male... by gwern on Wednesday November 22, 2017 @11:00AM (#15757762)
You can run A/A studies in the sense of permutation tests or bootstraps where you shuffle individual mice which weren't treated and see how often you get a 'significant' result; this is how you can see stuff like litter or cage or room effects (ie mice housed close to each other will respond more similarly to each other, and woe betide you if you do your 'randomization' by selecting from different cages rather than selecting from different individuals!) eg "Design, power, and interpretation of studies in the standard murine model of ALS" http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.654... , Scott et al 2008, subsamples data from their 'control' mice to empirically estimate pseudo-effects and the null distribution. This is equivalent to raising two batches of mice and comparing. More interesting are multi-lab attempts at eliminating that heterogeneity and getting consistency in results, the most recent example being _C. elegans_ life extension - it turns out to be incredibly hard to harmonize labs in terms of measurements, environment, and results, and that's for well-established effects in model organisms which are about as simple as possible.

Unsurprisingly, there's a ton of heterogeneity which means it's easy to violate your statistical assumptions (i.i.d. being more important than normality here, I think) and so you get a lot of false positives. It's similar to the brain imaging issue: the assumption of spatial locality and Gaussianness is not really true and drives a lot of false-positive _p_-value inflation. There are really quite a lot of problems with animal studies, so it's not a surprise that they wipe out spectacularly when transferring to human.

I keep a list of systematic reviews & meta-analyses I run across at https://www.gwern.net/DNB-FAQ#fn95 One of the most impressive ones, I think, are sepsis treatments: ~150 sepsis treatment experiments in humans based on mouse/rat experiments, and all of them failed: "Genomic responses in mouse models poorly mimic human inflammatory diseases", Seok et al 2013 http://www.pnas.org/content/110/9/3507.full.html

Re:Ketamine lifts rodents' mood only if administered by male... by andai on Wednesday November 22, 2017 @03:29AM (#15755241)
Do you remember the article(s) in which he talks about that, or any other keywords? I found all of "In the Pipeline", but my Google-fu ends there.

Also for the mice getting fatter over time? That's fascinating!

Re:Ketamine lifts rodents' mood only if administered by male... by allenz on Wednesday November 22, 2017 @03:34AM (#15755264)
Re:Ketamine lifts rodents' mood only if administered by male... by gwern on Wednesday November 22, 2017 @10:40AM (#15757604)
Re:Ketamine lifts rodents' mood only if administered by male... by gwern on Wednesday November 22, 2017 @10:37AM (#15757568)
http://blogs.sciencemag.org/pipeline/archives/2015/08/21/is-...

Well worth reading through the archives, IMO, even if you're not a chemist or biologist (I'm not). Quite aside from the methodological issues and the realities of working in the pharmacorp industry, Lowe's "Things I Won't Work With" series about dangerous chemical compounds is... a blast.