Re:Cancer reproducibility effort faces backlash
by nycticorax on Saturday June 27, 2015 @04:56PM
(#9791510)
"it was trivial to randomize the parameters within their bounds of experimental measurement and create chaotic systems. Did this mean the model wasn't so great? No, it just meant those couldn't be the real-life configuration of those parameters . . . sigh."
Why is this sigh-worthy? Maybe the model is still ok. Maybe some of the parameters are constrained in nature in ways that you/we just don't know about yet. Of course, it could be that the model is bad, but you act as if your results, as stated, demonstrate this. Which they do not, as far as I can see.
Re:Cancer reproducibility effort faces backlash
by TisButMe on Saturday June 27, 2015 @03:15PM
(#9791226)
I hoped that it wasn't as bad in computational biology, or ecology, or any other biology field where systems and models are actually defined. It saddens me to read that your experience was as bad as mine...
Re:Cancer reproducibility effort faces backlash
by donovanr on Saturday June 27, 2015 @03:30PM
(#9791267)
I'm a CompBio PhD student, and my experience is that folks in that field are much more careful with statistics than in, say, molecular biology labs, but it varies from lab to lab. My PI is exceedingly meticulous about stats -- for instance, we don't report p-values, but rather entire distributions -- but that's because our work is all in silico, so it's easy to run tons of replicate simulations. Wet lab work that's finicky should definitely be held to high statistical standards, but I don't think it's fair to presume everyone in the field guilty until proven innocent.
Re:Cancer reproducibility effort faces backlash
by TisButMe on Saturday June 27, 2015 @04:00PM
(#9791356)
But wet lab scientists should be even more careful! They have way less control over the system they're trying to study than you do, so stats are the only security net we have to even attempt to do anything with the data we produce.
I also agree on the innocent until proven guilty part, but by now I've seen and talked to hundreds of people with the best intentions, who do not realise how important careful examination of the data is, so I'm growing a bit disillusioned.
Re:Cancer reproducibility effort faces backlash
by peterfirefly on Sunday June 28, 2015 @02:27PM
(#9794466)
I think it is entirely fair to presume biologists incompetent and sloppy until they have proven otherwise.
(My impression from admittedly limited contact with biology students and from browsing through the occasional paper is that most of them barely approach mediocrity from below.)
I remember one professor studied metabolic reaction networks modeled with differential equations. The networks themselves were oversimplifications and relied on ~5N parameters (N being the number of compounds in the network). The problem was that while all the examples in publication converged on a nice steady state (yay homeostasis is a consequence of the model!) it was trivial to randomize the parameters within their bounds of experimental measurement and create chaotic systems. Did this mean the model wasn't so great? No, it just meant those couldn't be the real-life configuration of those parameters . . . sigh. And now I'm a data engineer and no one asks me to get data from an API that doesn't actually provide it and I'm much happier.