News for hackers. Stuff that matters.
Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor... by lmm on Monday August 18, 2014 @06:57AM (#8192121)
There are places that do that; at my first job we knew that the ETL code was core business functionality, and version-controlled it appropriately. And we knew that our ability to store raw data reliably was a competitive advantage.

But in many cases it's a technical debt issue. In the early startup stages when you're struggling for product-market fit, cleaning or auditing your data is not a good use of time. When a business like that starts to get traction, it finds itself with clients that now want more reliability, scaling issues, and a bunch of data that no-one really remembers where it all came from or how it was generated. A lot of the big-data startups are hitting that stage now.

There are no comments on this comment yet.