News for hackers. Stuff that matters.
Re:Show HN: We wrote a book to help data people build scalab... by khaito24 on Friday June 12, 2020 @11:48AM (#23500403)
Great book and spot on of the problem statement. One interesting note to point out is I didn't see any mention of testing your data models or version control? This should be part of the process of the modern analytics stack to ensure data quality.

A suggestion of approaches and tools could be useful. Whether it's via tools such as dbt for expected field values or with frameworks such as GreatExpectations. What happens to data that doesn't conform to expected values? How should you handle it? and how can the testing process be automated? This forms an important part of ensuring data quality and reliability of the analysed output.

Re:Show HN: We wrote a book to help data people build scalab... by huy on Friday June 12, 2020 @11:59AM (#23500512)
Yes.. we did get a number of early feedback about this exact topic of data quality. But we eventually decided to cover it in another separate sidebar to the book. It’s also not a simple topic to cover. And the book is supposed to “give you enough to be dangerous”.