News for hackers. Stuff that matters.
Re:There is no Data Engineering roadmap by xmprt on Thursday July 13, 2023 @10:14PM (#36718649)
> Data engineering can be unsatisfying if you thrive on writing reliable systems

This is what I needed for me to stop considering data engineering. Not that I was thinking about it a lot. I feel very happy with my current job but the thought of the code I write being so brittle as to paralyze me from making any changes terrifies me.

Re:There is no Data Engineering roadmap by quickthrower2 on Friday July 14, 2023 @12:35AM (#36719525)
Unless you get to work at the place that values your test driven perspective. Either because they are doing it or desperately want to (where want = they are already investing heavily in it)
Re:There is no Data Engineering roadmap by camgunz on Friday July 14, 2023 @05:55AM (#36721746)
We build pipelines in staging schemas, and we only promote them to prod once they pass tests. This is pretty analogous to SWE (it's where we got the idea) and is a pretty crucial part of us meeting the requirement where we don't put inaccurate data in front of customers. It's--of course--only as good as the tests, but that's every reliable system.