Re:For ‘Big Data’ Scientists, Hurdle to Insights Is ‘Janitor...
by lmm on Monday August 18, 2014 @06:57AM
(#8192121)
There are places that do that; at my first job we knew that the ETL code was core business functionality, and version-controlled it appropriately. And we knew that our ability to store raw data reliably was a competitive advantage.
But in many cases it's a technical debt issue. In the early startup stages when you're struggling for product-market fit, cleaning or auditing your data is not a good use of time. When a business like that starts to get traction, it finds itself with clients that now want more reliability, scaling issues, and a bunch of data that no-one really remembers where it all came from or how it was generated. A lot of the big-data startups are hitting that stage now.
But in many cases it's a technical debt issue. In the early startup stages when you're struggling for product-market fit, cleaning or auditing your data is not a good use of time. When a business like that starts to get traction, it finds itself with clients that now want more reliability, scaling issues, and a bunch of data that no-one really remembers where it all came from or how it was generated. A lot of the big-data startups are hitting that stage now.