News for hackers. Stuff that matters.

A comment on Shopify's Data Science and Engineering Foundations (2020).

Re:Shopify's Data Science and Engineering Foundations (2020) by snidane on Friday March 11, 2022 @09:26PM (#30648416) 

  > We test every situation that we can think of: errors, edge cases, and so on.
Pretty bold claim. Either their fantasy can't think of many weird data issues, or their data is somehow guaranteed to be always validated, or they came up with a magic secret sauce for data unit testing.

Could anybody explain what they mean by their version of unit testing?

Does it mean to test that a change in code will still successfully process the same static data sample as was tested previously?

If not using static data, what does it test when the input data changes? Does it validate the input data?

[ Parent | on HN ]
Re:Shopify's Data Science and Engineering Foundations (2020) by dafelst on Friday March 11, 2022 @10:36PM (#30648784)
I don't know what they do at Shopify, but I have seen teams be successful with using dbt's (https://getdbt.com) unit testing methodologies and tooling. It makes it very easy to do things like cardinality and null checking, and if you're using a datamart like Snowflake with snapshots, it is super easy to have test harnesses with representative data. To me the combination of tools there is the secret sauce, since data mocking is such a giant tedious pain in the ass.
[ Parent | on HN ]
Re:Shopify's Data Science and Engineering Foundations (2020) by croes on Saturday March 12, 2022 @02:55AM (#30649786)
>We test every situation that we can think of

Doesn't say anything about how many situations they think of.

[ Parent | on HN ]
Re:Shopify's Data Science and Engineering Foundations (2020) by burnoutgal on Saturday March 12, 2022 @08:54AM (#30651362)
When I was at Shopify, it was Python unit tests for ETL code. There was a framework of reusable components that simplified things, but it was all homegrown. They were trying to move to dbt
[ Parent | on HN ]
Re:Shopify's Data Science and Engineering Foundations (2020) by nojito on Saturday March 12, 2022 @08:09AM (#30651092)
Validating data on ingest solves many many problems down the line.
[ Parent | on HN ]