News for hackers. Stuff that matters.
Re:Data engineering and software engineering are converging by CalRobert on Friday August 29, 2025 @04:19PM (#45068927)
Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheapened.
Re:Data engineering and software engineering are converging by kentm on Friday August 29, 2025 @07:55PM (#45070714)
Yep, I specifically asked my company to make sure my job title was not “data engineer” when working on data infrastructure, because there was a growing trend of using it to mean “can write some sql”.

Likewise, we had to steer HR away from “data engineer” because we got very mixed results with candidates.

Re:Data engineering and software engineering are converging by itsoktocry on Friday August 29, 2025 @08:31PM (#45070899)
Ironic, since "Data Engineers" are probably far more in demand right now than "Software Engineers".
Re:Data engineering and software engineering are converging by ozim on Friday August 29, 2025 @09:39PM (#45071190)
Only in places silly enough to believe software devs/enga cannot write SQL.
Re:Data engineering and software engineering are converging by majormajor on Saturday August 30, 2025 @12:33AM (#45071937)
The "write SQL for ETLs all day" job is a risky one right now since LLMs really lower the barrier for dealing with gnarly SQL. So it's still not a bad time to have your resume be as clear as possible that you're the "deals with complex distributed systems" SWE type instead.
Re:Data engineering and software engineering are converging by mrugge on Saturday August 30, 2025 @01:03AM (#45072033)
That's more of an analytics engineer role. LLMs lower the barrier to entry, but popular SQL queries are about correctness and flexibility and this often requires deep understanding and ownership of each filter and window function. This lower barrier can quickly can turn into enough rope to hang yourself.
Re:Data engineering and software engineering are converging by CyberDildonics on Saturday August 30, 2025 @09:26AM (#45074476)
Why would a software engineer not be able to do both roles?
Re:Data engineering and software engineering are converging by omgwtfbyobbq on Sunday August 31, 2025 @08:07PM (#45088249)
They can, but it takes time away from software engineering.

Along the same reason, that's why there are DBAs, dev-ops engineers, etc...

Re:Data engineering and software engineering are converging by sdairs on Friday August 29, 2025 @04:38PM (#45069124)
I think even before dbt turned DE into "just write sql & yaml", there was an appreciable difference in DE vs SE. There was defo some DEs writing a lot of java/scala if they were in Spark heavy co's, but my experience is that DEs were doing a lot more platform engineering (similar to what you suggest), SQL and point-and-click (just because that was the nature of the tooling). I wasn't really seeing many DEs spending a lot of time in an IDE.

But I think whats interesting from the post is looking at SEs adopting data infra into their workflow, as opposed to DEs writing more software.

Re:Data engineering and software engineering are converging by craneca0 on Friday August 29, 2025 @05:16PM (#45069488)
yeah, i've seen large fortune 100 data and analytics orgs where the majority of folks with data engineering titles are uncomfortable with even the basics of git.
Re:Data engineering and software engineering are converging by vjvjvjvjghv on Friday August 29, 2025 @07:33PM (#45070572)
We have these at my company. They refuse to do any infrastructure work so you have to spoon feed the databases to them ready to go. It’s pretty annoying.
Re:Data engineering and software engineering are converging by Foobar8568 on Friday August 29, 2025 @11:29PM (#45071678)
Or the basics of SQL...
Re:Data engineering and software engineering are converging by isaacremuant on Friday August 29, 2025 @05:08PM (#45069402)
Agreed. Weird distinction to pay less to people who did certain things and you could a high variance between "data engineers". Some who had only done a course and others that had extensive knowledge of software engineering practices were considered the same.

Ridiculous.

Re:Data engineering and software engineering are converging by mandeepj on Saturday August 30, 2025 @12:26AM (#45071923)
> Data engineering was software engineering from the very beginning.

There it is! I found the post title was strange. Thanks for setting the record straight so succinctly.

Re:Data engineering and software engineering are converging by majormajor on Saturday August 30, 2025 @12:31AM (#45071935)
"Data Engineering" being considered a different role from "regular" SWE predates DBT by... at least one decade? If not two? Probably folks working with Hadoop vs RDMS DBA jobs.
Re:Data engineering and software engineering are converging by snthpy on Saturday August 30, 2025 @05:20AM (#45073212)
In work yes, but as a title? I only started seeing it called that around dbt origin.
Re:Data engineering and software engineering are converging by mrugge on Saturday August 30, 2025 @01:20AM (#45072078)
Titles in software engineering have never mattered less than they do today. Energy worrying about titles or jealosy over specific tech ownership is best channeled into focus on customer, on problem to solve and on finding the best way to solve it as a team.
Re:Data engineering and software engineering are converging by omgwtfbyobbq on Saturday August 30, 2025 @04:09PM (#45077618)
Part of the problem is that a BA/BSA who writes Python, SQL, etc... as part of their day to day work will get lumped with those who don't and their salary doesn't reflect their skills and work product.

Obviously the same can apply in any given title, and does with data engineers like you pointed out, but it's not as simple as just title inflation.