Re:Show HN: We wrote a book to help data people build scalab...
by huy on Friday June 12, 2020 @11:59AM
(#23500512)
Yes.. we did get a number of early feedback about this exact topic of data quality. But we eventually decided to cover it in another separate sidebar to the book. It’s also not a simple topic to cover. And the book is supposed to “give you enough to be dangerous”.
A suggestion of approaches and tools could be useful. Whether it's via tools such as dbt for expected field values or with frameworks such as GreatExpectations. What happens to data that doesn't conform to expected values? How should you handle it? and how can the testing process be automated? This forms an important part of ensuring data quality and reliability of the analysed output.