News for hackers. Stuff that matters.
Re:The Evolution of the Data Engineer Role by abrazensunset on Monday October 24, 2022 @05:12PM (#33322735)
There is a huge round of "data observability" startups that address exactly this. As a category it was overfunded prior to the VC squeeze. Some of them are actually good.

They all have various strengths and weaknesses with respect to anomaly detection, schema change alerts, rules-based approaches, sampled diffs on PRs, incident management, tracking lineage for impact analysis, and providing usage/performance monitoring.

Datafold, Metaplane, Validio, Monte Carlo, Bigeye

Great Expectations has always been an open source standby as well and is being turned into a product.

Re:The Evolution of the Data Engineer Role by mywittyname on Tuesday October 25, 2022 @03:41PM (#33335200)
Thanks for the recommendations, I'm going to check some of them out.