News for hackers. Stuff that matters.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by mrdmnd on Monday August 18, 2014 @12:03AM (#8191381)
My team at Airbnb is working on a tool called "Salus" - named after the Roman goddess of hygiene - that allows you to check the correctness of your data in a number of common, extensible ways. Perhaps this is something that the open-source community at large might be interested in?
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by jcdreads on Monday August 18, 2014 @09:32AM (#8192511)
I have rolled an extensive (purpose-built) general data quality system at work, and would love for there to be a general purpose tool that does the same sort of thing. I would love to use, contribute to, or otherwise make real an open source version of such a tool.

There is an enormous need in several industries for this sort of thing; but most people don't really know they need it yet.

Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by rgejman on Monday August 18, 2014 @12:04AM (#8191383)
Interesting. I'd love to hear the details.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by palewire on Monday August 18, 2014 @09:06AM (#8192435)
Yes. This is something that could be useful to data journalists, who often have to quickly grip and rip new datasets.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by christopheraden on Monday August 18, 2014 @06:17AM (#8192035)
I'd be interested. Any tool that checks correctness is one less thing I have to write before I begin my analyses.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by pbnjay on Monday August 18, 2014 @08:57PM (#8195539)
Sounds intriguing... Is it something rule-based where I have to define things ahead of time, or can it infer stuff - for example highlighting outliers in an automatically detected numeric field?
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by ehurrell on Monday August 18, 2014 @10:52AM (#8192874)
Definitely something I'd be interested in taking a look at, I imagine others would as well!
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by seancron on Tuesday August 19, 2014 @12:51AM (#8196063)
Absolutely. Would definitely be interested in hearing more about it.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by twone2 on Monday August 18, 2014 @10:47AM (#8192857)
Absolutely - please keep me updated on progress!
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by joncooper on Monday August 18, 2014 @01:58PM (#8193684)
Yes.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by hmason on Monday August 18, 2014 @09:05AM (#8192432)
yes.