News for hackers. Stuff that matters.
Re:Gene name errors are widespread in the scientific literature by viraptor on Wednesday August 24, 2016 @02:02AM (#12350014)
I did some small tasks for people working in bioinformatics and what I've seen is both amazing (the science part) and terrifying (the tools). Apparently I saved hours of retyping for one person who was doing a manual JOIN on gene names between two CSV files. As in ctrl+f name of gene from file A, copy, paste into another window. For thousands of rows. I was trying to explain that you can import CSV files without autoformatting as well, but they didn't believe me... sigh... (at least they were aware that this is a bad idea with default settings, because the formatting changes)

I don't know if this can be fixed and how. Lots of people seem to have their own process and they neither understand that the tools can be used more efficiently / correctly, nor question the long, manual process they follow currently. They basically require an "intermediate excel" course which ends with "if you're doing copy/paste 3 times in a row, you should look for better solution". I'm not even questioning the use of excel at this point...

Re:Gene name errors are widespread in the scientific literature by joshvm on Wednesday August 24, 2016 @04:35AM (#12350590)
This kind of problem is rife in academia.

Part of the problem, as a sibling mentioned, is that many people get into these kinds of fields from non-programming or even computer-illiterate backgrounds. At an undergraduate level, even in heavily numerical disciplines like physics, there is relatively little coding. Even then, it's tedious crap like F90 and bulletproof C. So people get into a Masters or a PhD in a field they love and are suddenly confronted by data analysis, and they have no idea what to do.

I've spoken a lot to my girlfriend (an astrophysicist) about this, as she's in this position herself. It's not that she isn't smart, she just has little to no experience with data wrangling and she's (in my opinion) been inadequately trained. I've solved things with Python one-liners that she would have spent literally days doing manually. We've had conversations along the lines of:

"But why does this file have 700 lines of input data filenames hardcoded? You know you could write some code to grab them for you?"

"Yes, but in the time it would take me to write the code, I may as well do it by hand." [In the end I wrote a 5 line regex to do it]

So I can assure you, people question the manual process and they think you're magical when you show them a quicker way of doing things. However, often there isn't the motivation or confidence to try and be magical themselves.

Re:Gene name errors are widespread in the scientific literature by Agentlien on Wednesday August 24, 2016 @05:16AM (#12350717)
My wife works at the virology lab of a local hospital. They do a lot of validation and quality controls. However, when she started there, all their statistics were calculated by hand using approximate equations which were supposed to make it doable using a simple calculator. Whoever had set this up lacked a strong understanding of statistics. Still, this procedure had remained unchanged since the eighties and no one knew how or why it was supposed to work. Going back in old records, my wife found that there were a lot of errors in the recorded results.

She spent a while setting up a simple Excel sheet into which they enter all test results and now all the statistics are calculated automatically using proper methods.

Re:Gene name errors are widespread in the scientific literature by SixSigma on Wednesday August 24, 2016 @07:54AM (#12351327)
> This kind of problem is rife in academia.

This kind of problem is rife everywhere. People don't know and are not interested in how to use computers to help them because "it's too complicated".

Excel doesn't help.

Re:Gene name errors are widespread in the scientific literature by throwanem on Wednesday August 24, 2016 @06:57AM (#12351066)
Is this necessarily a bad thing entire? House wizard can be a fairly comfortable position.
Re:Gene name errors are widespread in the scientific literature by th0waway on Wednesday August 24, 2016 @01:35PM (#12353725)
This is where I take the time to explain to everyone that "Yeah, while it may save you time to do it manually, if you do it THIS way instead, you never have to write it from scratch again, and your output goes waaay up".

Every job I've ever had, I always look at how my predecessors do the job, then automate the crap that I don't want to deal with. :/

Re:Gene name errors are widespread in the scientific literature by joshvm on Thursday August 25, 2016 @04:52AM (#12357682)
I'm in the middle of writing a retrospective on my PhD - the stuff I wish I'd done at the beginning. Perhaps unsurprisingly, it mostly boils down to automate and pipeline everything. It almost always saves time because inevitably you have to repeat work with parameters changed very slightly.
Re:Gene name errors are widespread in the scientific literature by drauh on Wednesday August 24, 2016 @02:37AM (#12350165)
The prevailing attitude is "I want to do biology, not computer science". I find it pretty frustrating. This is probably one of the last fields of science to be seriously computerized.
Re:Gene name errors are widespread in the scientific literature by alephnil on Wednesday August 24, 2016 @02:34AM (#12350154)
I used to work as a bioinformatics programmer, and in some cases I did some processing in an afternoon, including writing a python script, that a postdoc had used a month doing manually in Excel.

While the tools often are crude, a problem is that biochemical and molecular biology researchers lack the bioinformatics skills and other IT skills that often is badly needed in modern genetic research. The core competence of these researchers is in the lab, and bioinformatics skills is outside of that. That you see effects like conversion of gene names in excel does not surprise me at all.

You also see the same problem in other areas where researchers need skills outside of their usual comfort zone. The classical example is statistics.