Re:There is no Data Engineering roadmap
by yesiamyourdad on Friday July 14, 2023 @10:21AM
(#36724411)
Maybe it's just the positions I've ended up in, but my DE roles have required a TON of ops as well. You can build all these pipelines, but you will be absolutely buried in operational work if you can't get the automation squared away. With the role I've been in for the last 2 years, the most significant things we've done were done in Terraform, Ansible, and Prometheus. Sure, there's lots of SQL and Python, some Java, some Scala, but the biggest issue has been getting the system to where we get useful and actionable alerts and downstream data consumers get notified about late or missing data without requiring manual intervention from the data engineers.
[ Parent
| on HN ]
Agree with this. I think a good 75% of the value I've delivered in data engineering roles has been on proficiency building out cloud infrastructure, CICD, etc.
Teams hire data engineers when the analysts are spending too much time and effort (and failing to succeed at) building data pipelines. Often there is a significant need for engineering and cloud infrastructure improvements beyond fixing up a few wonky python ETLs and broken SQL queries.
I'll second this too. As a former manager told me, figure out how to scale yourself. Making the end to end development lifestyle easy for peers and partner teams is one way to do that. It could just as easily be called platform or systems engineering with a touch of DevOps.
It's like UX for data people. How to make a cohesive experience among the various tools, scripts, services that people use day-to-day so they can use and maintain datasets efficiently.