News for hackers. Stuff that matters.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by rgejman on Sunday August 17, 2014 @11:50PM (#8191362)
Data cleaning/munging is a surprisingly pervasive problem in my field ('big data' cancer research). I spend a good fraction of my time hunting down bugs caused by dirty sources upstream of my scripts. One way to fix this is to become more rigorous about using tools and libraries that check and enforce data correctness and consistency... but it's hard to do this on an individual level. Even if I'm careful about ensuring the correctness of my data, other researchers who send me data may not be as scrupulous. As with any challenge that requires teamwork and constant vigilance (e.g. unit testing), perhaps the data science community needs to invest in tools and computational frameworks that constantly monitor data for correctness and consistency using DDLs and the like.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by mrdmnd on Monday August 18, 2014 @12:03AM (#8191381)
My team at Airbnb is working on a tool called "Salus" - named after the Roman goddess of hygiene - that allows you to check the correctness of your data in a number of common, extensible ways. Perhaps this is something that the open-source community at large might be interested in?
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by jcdreads on Monday August 18, 2014 @09:32AM (#8192511)
I have rolled an extensive (purpose-built) general data quality system at work, and would love for there to be a general purpose tool that does the same sort of thing. I would love to use, contribute to, or otherwise make real an open source version of such a tool.

There is an enormous need in several industries for this sort of thing; but most people don't really know they need it yet.

Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by rgejman on Monday August 18, 2014 @12:04AM (#8191383)
Interesting. I'd love to hear the details.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by palewire on Monday August 18, 2014 @09:06AM (#8192435)
Yes. This is something that could be useful to data journalists, who often have to quickly grip and rip new datasets.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by christopheraden on Monday August 18, 2014 @06:17AM (#8192035)
I'd be interested. Any tool that checks correctness is one less thing I have to write before I begin my analyses.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by pbnjay on Monday August 18, 2014 @08:57PM (#8195539)
Sounds intriguing... Is it something rule-based where I have to define things ahead of time, or can it infer stuff - for example highlighting outliers in an automatically detected numeric field?
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by ehurrell on Monday August 18, 2014 @10:52AM (#8192874)
Definitely something I'd be interested in taking a look at, I imagine others would as well!
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by seancron on Tuesday August 19, 2014 @12:51AM (#8196063)
Absolutely. Would definitely be interested in hearing more about it.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by twone2 on Monday August 18, 2014 @10:47AM (#8192857)
Absolutely - please keep me updated on progress!
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by joncooper on Monday August 18, 2014 @01:58PM (#8193684)
Yes.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by hmason on Monday August 18, 2014 @09:05AM (#8192432)
yes.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by kiyoto on Monday August 18, 2014 @12:25AM (#8191429)
If cancer research is anything like high frequency trading and time series analysis (which I used to work on), then one heinous thing about corrupted data is inter-dependencies among different data processing tasks: if data coming in "early" in the data workflow topology is garbage, its errors are propagated throughout the dataflow, resulting in multiple days of data detective work, laborious undoing of everything that went wrong, etc.

Also, I observe that the data science community does not talk about these issues nearly as much as they should either because 1) it doesn't make a very inspirational topic 2) they are shielded from this kind of issue by a data engineering team.

Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by rgejman on Monday August 18, 2014 @08:16AM (#8192297)
Agreed. We need to talk about these issues as a community and come up with effective strategies to prevent data contamination. We've done this for web development and the result was a panoply of tools to validate form input. Now we need to do this for our back-end tools.

For instance, statistical programming languages like R, which generally operate on nice tabular data, should come with built in methods to enforce data validity. I should be able to tell R that I expect certain columns to only contain values from 0-1 and no NAs. This is an easy example because languages like R are somewhat dictatorial about how they want you input and process data, but one can imagine the same sort of methods built into the base libraries of general purpose languages.

On the other hand, maybe we need to be more rigorous on the whole about data validation. We accept that automated/continuous testing is an effective mechanism for preventing bugs. We need the same automated systems to check data files and flag them when there are issues.

Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by hadley on Monday August 18, 2014 @12:07PM (#8193225)
Take a look at the testdat package which provides some tools in the direction.