Re:The Forgotten Job of a Data Scientist: Editing
by eshvk on Sunday May 11, 2014 @03:33PM
(#7729662)
> There are plenty of "data scientists" who use canned tools and play around with parameters because that's all "the business" thinks it needs.
As with a software engineer, it is a role that is different in every place. Every place has its own definition of the role. This is not bad. It is a mere reflection of the market conditions where there are a lot of people are simultaneously bad at Linear Algebra, Probability and Statistics and dangerous enough to write production code fast. (Your standard C.S. grad SWE).
There are plenty of "data scientists" who use canned tools and play around with parameters because that's all "the business" thinks it needs.
You want to trim complexity for a reason that any data scientist worth his salt (and there are plenty of celebrity engineers in SF making $500k who aren't worth their salt and don't know this) should already know: bias-variance tradeoff (see also: underfitting and overfitting). If your model is too flexible/complex, it will begin absorbing noise. That leads to a model that performs extremely well on training data but fails miserably on unseen data. There are well-studied techniques for preventing this, but I'd guess that fewer than 20% of self-described or titled "data scientists" are familiar with them.