News for hackers. Stuff that matters.
Re:Data Engineering Design Patterns (DEDP) by snapetom on Thursday December 07, 2023 @11:44PM (#38565558)
This lines up with my experience, and I've found it heavily depends on what industry you are in.

> software engineer in the data domain.

I was basically this for the past ten years. Maybe it was because I was working only in startups.

Outside of tech/startup orgs, "data engineer" at least I found, were SQL specialists. About six years ago, I went into healthcare, and discovered there were about 30 people across five teams that were data engineers. "Oh cool. My colleagues," I thought. Imagine my surprise when I found they only knew SQL, knew data modeling theory, and had basically no SDLC experience. At my present job, in a traditionally blue-collar industry, I took over a team with the only data engineer in the whole company. He, too, knew only SQL. I've had shove Python at him and get him working in SDLC.

I think these people though, are shrinking. Putting pressure on this from the other side, Python is a common skill with data analyst these days. Software engineers do the heavy lifting and good-enough data modeling, while data analyst do business-specific analysis and good-enough software development like writing DAGs with Dagster. Knowing SQL isn't enough to get by in the job market.

Re:Data Engineering Design Patterns (DEDP) by desert_rue on Friday December 08, 2023 @12:03AM (#38565653)
I know plenty of people who only use sql. There is now a role called analytics engineer that primarily sql and often with transformation tools like dbt.

I personally haven’t met a lot of software devs who would call their data modeling capabilities ‘good enough’. It is a different way of thinking to go from building 3rd form normalized tables to denormalizing everything. Plus if a company is merging data from many systems, it makes it more complicated for the software developer.

Re:Data Engineering Design Patterns (DEDP) by fifilura on Friday December 08, 2023 @02:07AM (#38566211)
The moment you have learned SQL, is when you have forgotten everything about C/C++