Re:We don't need data scientists, we need data engineers
by Jugurtha on Friday January 15, 2021 @06:52AM
(#25789798)
>Hell, in Slack, I watch their data scientists continuously wrestle with how to actually make their Jupyter notebooks work in production.
Could you go into more details on what their struggles are? We had many problems as a company doing machine learning projects, and we built our internal platform (https://iko.ai) to keep our sanity. I'm always interested in problems others may be having.
Re:We don't need data scientists, we need data engineers
by EdwardDiego on Friday January 15, 2021 @07:03PM
(#25798342)
Python hell, basically. Getting the right Python version, right dependency versions etc.
Re:We don't need data scientists, we need data engineers
by Jugurtha on Friday January 15, 2021 @09:08PM
(#25799252)
We didn't have it any better wit Scala. We'd run sbt and it would download the internet.
Our belief was that there are some odd behaviors in every tool and we had to figure out a way around.
And, their production alleged "machine learning" (it's pretty much standard linear regression, but calling it ML is sexy) systems are slow motion train-wrecks. If the string and duct tape holds, then it works, but it's unfortunately continually breaking.
Hell, in Slack, I watch their data scientists continuously wrestle with how to actually make their Jupyter notebooks work in production.
Whereas my company has far more data engineers than data scientists. The plan from higher up the corporate food chain was always that we'd give them our data to do their data science voodoo on, but we ended up getting a few data scientists of our own for specific projects.
So, we focused on ensuring our data stream was reliable, consistent and sufficiently timely, for them to work on. But as soon as it hits their systems, it's a forest fire of hacks upon hacks, which inevitably break.
In the end, we had to send in our data engineers to stabilise their flagship "real time reporting" product that corporate was so amped about.
So yeah, I think that there's probably a happy ratio of data scientists to data engineers, of about, say, 1:5 or 1:10, because the maths generally scales O(1), it's the beauty of maths, but the actual engineering to get clean data delivered timely without breaking anything scales very differently indeed.