News for hackers. Stuff that matters.
Re:Goodbye, Data Science by jstx1 on Tuesday November 29, 2022 @08:50AM (#33787508)
As someone who also wants to move away from data science, data engineering is the last thing I would want to do. I think DE comes with many of the same problems and it's also a very ill-defined career track; I wouldn't recommend it to anyone. ML engineer or backend developer seem like much more appealing job profiles.
Re:Goodbye, Data Science by aulin on Tuesday November 29, 2022 @09:04AM (#33787687)
would you highlight some of the biggest differences between ML engineering and data engineering? I believe they're sometimes used interchangebly especially if "data" is "datasets" for ML.
Re:Goodbye, Data Science by apohn on Tuesday November 29, 2022 @09:26AM (#33787957)
IME, this is the difference.

Data Engineers are the people who take raw data (e.g. what lands in S3) and put that into data systems that can be used by other systems (e.g. Dashboards) and people (e.g. Analyst, Data Scientists, BI people). Data Engineers clean data, but they are really looking at cleaning out systemic issues (e.g. some data that is missing in one field is in another field, and that needs to be consolidated) and not the scrutinized row-by-row cleaning that Data Scientists end up doing. Data Engineers also do the data steps (e.g. creating a performant stored query) required to support things like business KPIs and reporting.

ML Engineering has a lot more variety based on the company and org, but generally it's about building an automated pipeline that includes ML. In smaller orgs you do everything - build a data pipeline, train a model, deploy that model, score new data, etc. In larger orgs, ML Engineers take a model built by somebody else and make it run at scale while meeting certain SLAs (e.g. making recommendations on a social media website).

Re:Goodbye, Data Science by jstx1 on Tuesday November 29, 2022 @09:13AM (#33787802)
Data engineers don't work with machine learning at all. In fact one of the reasons why it developed as a job title over time waas specifically to differentiate the people who work with data but don't do any statistics or ML. If a DE who is doing "datasets for ML" decides to call themselves an ML engineer, they're just getting a bit too creative with the job titles (maybe they want a career change, more money, they think it sounds better, all sorts of reasons).

As an ML engineer you might need to do some data engineering work as part of your job but not the other way around.

Re:Goodbye, Data Science by omega3 on Tuesday November 29, 2022 @09:21AM (#33787899)
> Data engineers don't work with machine learning at all.

This very much depend on the company. From experience DE is used as a catch-all title.

Re:Goodbye, Data Science by jstx1 on Tuesday November 29, 2022 @09:25AM (#33787936)
Data scientist is much more catch-all from what I've seen. But a lot of that varies a lot by geography too (for example in the US people very often use DS very differently from how the title is used in the UK).
Re:Goodbye, Data Science by pinum on Tuesday November 29, 2022 @04:10PM (#33793365)
>in the US people very often use DS very differently from how the title is used in the UK

I haven't heard about this before and now I'm curious- can you elaborate on the differences?

Re:Goodbye, Data Science by jstx1 on Tuesday November 29, 2022 @05:27PM (#33794262)
In the US it's more common for data scientist to be similar to a product analyst or a data analyst perhaps with better technical skills. In the UK data scientist is more likely to be someone who is doing applied ML work (other titles for this are ML engineer or applied data scientist).

Obviously it's not a perfectly clean separation but it's a trend, and people sometimes end up really talking past each other. You can see on r/datascience which is very US-heavy how people often recommend to beginners not to bother with advanced ML, stick to SQL, basic Python and analytics, and in the UK data science job market that's outright bad advice (it's fine advice for the UK analytics market which is a separate thing).

Re:Goodbye, Data Science by jghn on Tuesday November 29, 2022 @09:58AM (#33788360)
This. "Data Engineering" is pretty far from having a standard definition in the wild. If someone is describing a role as "data engineering" about the only thing you can count on being true is that it involves data.
Re:Goodbye, Data Science by tomrod on Tuesday November 29, 2022 @02:08PM (#33791747)
I haven't come across that. Unless the job title is being polluted like DS was (to include all aspects of data management), DE is specifically about data pipelines and not the models generating those clusters, predictions, or classifications.
Re:Goodbye, Data Science by adammarples on Tuesday November 29, 2022 @09:29AM (#33787997)
Depends. If their favourite data engineer says "Oh hey, I can write tensorflow too", then guess who get the job of to "productionizing" their crappy data science notebooks?
Re:Goodbye, Data Science by jstx1 on Tuesday November 29, 2022 @09:36AM (#33788072)
You have two much more likely options:

1. The person who developed the notebook is responsible for productionizing it. (No, it's not all crappy notebooks and some data scientists can indeed write high quality code).

2. You have someone like an ML engineer whose job it is to do this.

What you're describing seems like the least likely option; at least on the teams I've worked on "I can write tensorflow" would get you nowhere if that's not already a part of your job description.

Re:Goodbye, Data Science by adammarples on Tuesday November 29, 2022 @03:46PM (#33793054)
Well, anyways that's what I was doing this time last year. Much too small a team to have a dedicated ML engineer though.
Re:Goodbye, Data Science by zhdc1 on Tuesday November 29, 2022 @09:55AM (#33788336)
Somewhat surprised that there's a separate job category for what sounds like large-scale data cleaning and aggregation work (which IMO is 90%+ of the effort involved with data science).

Anyway, I'm going to go back to my 5K+ lines of code for an upcoming conference submission - almost all of which involve data cleaning and aggregation - and think about how I could be making a 2x more than I am now.

Thanks Hacker News.

Re:Goodbye, Data Science by boringg on Tuesday November 29, 2022 @09:42AM (#33788168)
Data Engineer = build the data pipelines; ML Engineer = implement the models
Re:Goodbye, Data Science by adamsmith143 on Tuesday November 29, 2022 @10:24AM (#33788665)
Not sure why this is confusing, sounds like people think they are the same because they have never worked in the area. It's always been very well defined at Corps I've been at.
Re:Goodbye, Data Science by starwind on Tuesday November 29, 2022 @10:25AM (#33788683)
I went data science to data engineering and been very happy. You may have trouble moving into backend development direct from data science because data scientists don't have a reputation for writing solid maintable code so data engineering could be a nice intermediate step cause it's heavy python
Re:Goodbye, Data Science by mynameisash on Tuesday November 29, 2022 @11:04AM (#33789254)
My title is still software engineer, but I effectively do data engineering, and I work closely with data scientists.

I love a lot of it, but there's still plenty of bullshit to deal with. Just in the technical side, dealing with Python is a perpetual gong show, and most of my team's work seems to revolve around configuration of secrets and K8s.

I'm fortunate to be the guy that nerds out about performant code, so when something inevitably turns out to be a perf bottleneck, I can turn back into a regular old software engineer who trades in big data. Which I think is a better title/charge than data engineer, anyway.

I've talked with plenty of ML engineers, and they seem to immensely enjoy what they do. It seems that the periphery of data engineering is great; the core of it, not so much.

Re:Goodbye, Data Science by throwaway2037 on Tuesday November 29, 2022 @11:22AM (#33789519)
You wrote: <<dealing with Python is a perpetual gong show>> What does this mean?
Re:Goodbye, Data Science by selimthegrim on Tuesday November 29, 2022 @12:28PM (#33790479)
The Gong Show was a 70s US game show notorious for its onscreen anarchy
Re:Goodbye, Data Science by andrewnc on Tuesday November 29, 2022 @01:03PM (#33790938)
I think "The Gong Show" was an old tv show about amateur talents. Sometimes good, most of the time terrible and hilariously unaware. Not sure if that was what was intended here.
Re:Goodbye, Data Science by throwaway2037 on Thursday December 01, 2022 @04:00AM (#33813963)
So is the GP criticizing Python? If yes, I am curious to know why. No, I am not here to defend Python. The constant runtime exceptions due to typing mistakes is so tiring.