News for hackers. Stuff that matters.
Re:Ketamine lifts rodents' mood only if administered by male... by jaggederest on Wednesday November 22, 2017 @02:48AM (#15755068)
I think it would be very interesting to have labs run detailed A-A studies to investigate these kind of things.

"This kind of mouse we raised under these conditions, and we changed nothing, but they varied in these ways over time"

I often wonder whether the variables we check against are actually as independent and subject to the central limit theorem as we assume.

Re:Ketamine lifts rodents' mood only if administered by male... by gwern on Wednesday November 22, 2017 @11:00AM (#15757762)
You can run A/A studies in the sense of permutation tests or bootstraps where you shuffle individual mice which weren't treated and see how often you get a 'significant' result; this is how you can see stuff like litter or cage or room effects (ie mice housed close to each other will respond more similarly to each other, and woe betide you if you do your 'randomization' by selecting from different cages rather than selecting from different individuals!) eg "Design, power, and interpretation of studies in the standard murine model of ALS" http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.654... , Scott et al 2008, subsamples data from their 'control' mice to empirically estimate pseudo-effects and the null distribution. This is equivalent to raising two batches of mice and comparing. More interesting are multi-lab attempts at eliminating that heterogeneity and getting consistency in results, the most recent example being _C. elegans_ life extension - it turns out to be incredibly hard to harmonize labs in terms of measurements, environment, and results, and that's for well-established effects in model organisms which are about as simple as possible.

Unsurprisingly, there's a ton of heterogeneity which means it's easy to violate your statistical assumptions (i.i.d. being more important than normality here, I think) and so you get a lot of false positives. It's similar to the brain imaging issue: the assumption of spatial locality and Gaussianness is not really true and drives a lot of false-positive _p_-value inflation. There are really quite a lot of problems with animal studies, so it's not a surprise that they wipe out spectacularly when transferring to human.

I keep a list of systematic reviews & meta-analyses I run across at https://www.gwern.net/DNB-FAQ#fn95 One of the most impressive ones, I think, are sepsis treatments: ~150 sepsis treatment experiments in humans based on mouse/rat experiments, and all of them failed: "Genomic responses in mouse models poorly mimic human inflammatory diseases", Seok et al 2013 http://www.pnas.org/content/110/9/3507.full.html