Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea?
by fifilura on Monday February 12, 2024 @12:19PM
(#39347529)
Whenever I get confronted with SWE best practices it is all about version control, unit tests, integration tests and CI/CD.
What are some good resources for the other things you describe?
Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea?
by jimberlage on Monday February 12, 2024 @12:42PM
(#39347839)
I don’t have great links handy, but read up on observability.
You’ll want to read up on what people do on error, especially in the case that contracts change (search “api versioning”.)
Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea?
by fifilura on Tuesday February 13, 2024 @05:48AM
(#39356464)
Thank you!
It may be that those topics are part of the SWE process but not generally applicable for example UI work and therefore overlooked.
But it may also be something else. That a process can create Grafana dashboards and processes for API updates like a pro.
But only the person that really cares about the data and know it from inside out can catch the "unknown unknowns". And for that you need to give access (and hand over responsibilities) to Analyst persons too and in some ways trust them too, even though they may not have as rigorous processes.
How you handle contract violations between dependencies and third parties is not an inherently different problem. What should you do if a data source changes schema? How do you detect it, alert, take action? How are users or customers or consumer jobs impacted? This is all just normal software.
In many ways it feels like you're making my case for me.