News for hackers. Stuff that matters.
Re:We don't need data scientists, we need data engineers by mynameisash on Thursday January 14, 2021 @11:32AM (#25778352)
> What ends up happening is you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team

Not to disparage the amazing data scientists I've worked with, but I've been on teams where this is very much the approach to operationalizing models. It's basically, "Here's the sklearn model and some fragile featurization scripts we built. Can you take this to prod ASAP?"

The problem I've seen is that DS & DE teams were in different parts of the org and had their own sprints that were in no way connected. So they kept chucking models over the wall and we kept trying to faithfully operationalize. Once we convinced leadership that we had to collaborate from the get-go, things went a whole lot better. It also improved the working relationship of engineers and scientists.

I learned a hell of a lot from the scientists; they learned how to write better code. They also learned what code they didn't need to write because I could do it faster or better than them, leaving them to focus on more important things. It was pretty amazing to find what manual processes they would setup in lieu of proper (or even any) engineering support. Again, these are amazingly smart people, but they were being square-pegged into a lot of round-hole engineering tasks.

Now, the much more frustrating issue I had was being in a very data-heavy organization and being told by a distinguished engineer (my skip-level) plus my direct manager that, "data engineering isn't a real discipline." I left that org very shortly thereafter.

Re:We don't need data scientists, we need data engineers by teraku on Friday January 15, 2021 @03:16AM (#25788279)
Not only that. If you have DS & DE in different orgs, often DE is in an IT org that also has to support legacy systems that sometimes become very time intensive. So then the DS org says they can't get stuff done because DE does not deliver, and DE can't get things done because the "elder engineers" in their org are not allowing reformation. So they are stuck between a rock and a hard place.
Re:We don't need data scientists, we need data engineers by civilized on Thursday January 14, 2021 @12:09PM (#25778981)
This is 100% my experience as a data scientist. The engineering support we get is restricted to submitting a ticket for database access or moving data from one system to another. Wouldn't dream of involving an engineer in a data science project team, because I have no evidence that they have any experience or expertise in anything other than tickets to move data around.
Re:We don't need data scientists, we need data engineers by Mauricebranagh on Thursday January 14, 2021 @12:40PM (#25779393)
That's first line support not engineering
Re:We don't need data scientists, we need data engineers by killtimeatwork on Friday January 15, 2021 @09:17AM (#25791180)
>> moving data from one system to another > That's first line support not engineering

Assuming the OP meant "setting up a pipeline for moving data from one system to another" and not a one-time copy, it is definitely engineering.

Re:We don't need data scientists, we need data engineers by civilized on Friday January 15, 2021 @10:14AM (#25791851)
Yeah, it's usually a pipeline
Re:We don't need data scientists, we need data engineers by mssundaram on Thursday January 14, 2021 @06:44PM (#25784733)
Yeah, because moving data around (which is hardly the entire responsibility of data engineer) is not useful at all