Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by mrdmnd on Monday August 18, 2014 @12:03AM
(#8191381)
My team at Airbnb is working on a tool called "Salus" - named after the Roman goddess of hygiene - that allows you to check the correctness of your data in a number of common, extensible ways. Perhaps this is something that the open-source community at large might be interested in?
[ Parent
| on HN ]
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by jcdreads on Monday August 18, 2014 @09:32AM
(#8192511)
I have rolled an extensive (purpose-built) general data quality system at work, and would love for there to be a general purpose tool that does the same sort of thing. I would love to use, contribute to, or otherwise make real an open source version of such a tool.
There is an enormous need in several industries for this sort of thing; but most people don't really know they need it yet.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by rgejman on Monday August 18, 2014 @12:04AM
(#8191383)
Interesting. I'd love to hear the details.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by palewire on Monday August 18, 2014 @09:06AM
(#8192435)
Yes. This is something that could be useful to data journalists, who often have to quickly grip and rip new datasets.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by christopheraden on Monday August 18, 2014 @06:17AM
(#8192035)
I'd be interested. Any tool that checks correctness is one less thing I have to write before I begin my analyses.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by pbnjay on Monday August 18, 2014 @08:57PM
(#8195539)
Sounds intriguing... Is it something rule-based where I have to define things ahead of time, or can it infer stuff - for example highlighting outliers in an automatically detected numeric field?
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by ehurrell on Monday August 18, 2014 @10:52AM
(#8192874)
Definitely something I'd be interested in taking a look at, I imagine others would as well!
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by seancron on Tuesday August 19, 2014 @12:51AM
(#8196063)
Absolutely. Would definitely be interested in hearing more about it.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by twone2 on Monday August 18, 2014 @10:47AM
(#8192857)
Absolutely - please keep me updated on progress!