We don't need data scientists, we need data engineers

Re:We don't need data scientists, we need data engineers by legerdemain on Thursday January 14, 2021 @06:22PM (#25784490)

Sure. So in a few years, an all-in-one 80% solution like Palantir Foundry will come along and there will suddenly be a lot less demand for data engineers.

Anecdotally, the former head of QA for Palantir UK is now the head of data engineering for Palantir UK, and Palantir does have an out-of-the-box, end-to-end, it-just-works product that handles 80% of ML workflows. You're betting your career that they won't put it in a box and sell it at commodity software prices?

[ Parent | on HN ]

Re:We don't need data scientists, we need data engineers by oivey on Friday January 15, 2021 @01:32AM (#25787700)

If something like that becomes mainstream, data scientists who just glue together canned ML algorithms from libraries should also be concerned. The kind of automation that enables eliding finding and cleaning the right data probably can also manage running PCA and basic ML.

[ Parent | on HN ]

Re:We don't need data scientists, we need data engineers by sgt101 on Friday January 15, 2021 @03:26AM (#25788349)

I remember the same claims for ISL Clementine before IBM bought it and turned it into SPSS modeler.

[ Parent | on HN ]