We don't need data scientists, we need data engineers

Re:We don't need data scientists, we need data engineers by hinkley on Thursday January 14, 2021 @01:34PM (#25780204)

This is a systemic problem. We ask non software engineers to write code, and then we expect them to apply a level of robustness and long term planning that even we have difficulty achieving. Not because we're being picky, but because we know the failure modes that are likely, and we know that people convince themselves that they aren't.

We've been through this with installer writers, database admins, test automation, operations people, and now 'devops' people who were supposed to be the answer to these problems. It never stops.

[ Parent | on HN ]

Re:We don't need data scientists, we need data engineers by AndrewKemendo on Thursday January 14, 2021 @02:05PM (#25780675)

It's a two way street. SWE need to learn some data practices and data folks need to learn some SWE practices.

[ Parent | on HN ]

Re:We don't need data scientists, we need data engineers by hinkley on Thursday January 14, 2021 @03:27PM (#25782017)

Oh absolutely. We'll build completely the wrong thing, but build it well (which just makes it all the harder to throw it away).

[ Parent | on HN ]