News for hackers. Stuff that matters.
Re:Data Engineering Design Patterns (DEDP) by kozikow on Thursday December 07, 2023 @02:37PM (#38560788)
It's the least defined role.

Currently, I am in a funny situation when all teams agree we need an additional data engineer. But basically:

- Sales and finance want more of business intelligence analyst

- Devs want more of a backend engineer

- ML researchers want a data analyst proficient in ETL to do pipelines on the training dataset

All of those 3 have only one thing in common - they need to know SQL very well. I've worked extensively with various technologies to analyze data - pandas, sql, spark. And still, I find SQL (especially recently BigQuery) getting me what I want the quickest.

Re:Data Engineering Design Patterns (DEDP) by bomewish on Sunday December 10, 2023 @02:09AM (#38589737)
I’m puzzled by this. Knowing SQL seems like a minor technical matter. You can learn whatever you need to do (or use gpt4 to quickly do stuff). Why does one need the expertise in syntax rather than just a general strong foundation in software and data stuff?