News for hackers. Stuff that matters.

A comment on We don't need data scientists, we need data engineers. Parent comment

Re:We don't need data scientists, we need data engineers by Jugurtha on Friday January 15, 2021 @06:52AM (#25789798)
>Hell, in Slack, I watch their data scientists continuously wrestle with how to actually make their Jupyter notebooks work in production.

Could you go into more details on what their struggles are? We had many problems as a company doing machine learning projects, and we built our internal platform (https://iko.ai) to keep our sanity. I'm always interested in problems others may be having.

[ Parent | on HN ]
Re:We don't need data scientists, we need data engineers by EdwardDiego on Friday January 15, 2021 @07:03PM (#25798342)
Python hell, basically. Getting the right Python version, right dependency versions etc.
[ Parent | on HN ]
Re:We don't need data scientists, we need data engineers by Jugurtha on Friday January 15, 2021 @09:08PM (#25799252)
We didn't have it any better wit Scala. We'd run sbt and it would download the internet.

Our belief was that there are some odd behaviors in every tool and we had to figure out a way around.

[ Parent | on HN ]