maartenatsoda

Early 2021, Tom Baeyens (creator of Activiti) and I started working on an open-source library to collect metrics and test data in SQL accessible tables (Apache 2.0). The goal was to make it easy for data engineers to catch data issues as early as possible in their data pipelines. We called it Soda SQL and our community of data engineers grew to over 1,000 Slack users and 35,000 downloads. Users liked Soda Core because of its easy of use, minimal dependencies, and SQL-first mentality.

In the first year, we kept on adding features to the library based on the input of hundreds of data engineers, dozens of which ended up contributing back to the project (thank you so much for being part of our journey). As a community, we added support for additional SQL dialects, introduced a programmatic interface, added operators for Airflow and Prefect, and make it easy to collect and store failed rows.

About 5 months ago, we felt like we were hitting some of the boundaries of the original scope of Soda SQL. We also wanted to introduce a new way to define checks on data, that was much more compact and human readable for everyone involved (including the non techies). Therefore, we decided to start from scratch and pour all our learnings into Soda Core. Today, our team is super excited to share with you that, after a good 2 months in beta, we’re ready to release Soda Core in GA.

Some of the key features of Soda Core are: (i) Human readable and writable checks (ii) Check data in Spark & SQL using fixed and dynamic thresholds (iii) 25+ built-in metric & check types (iv) Ability to manage check files as code

What’s next? (i) Support for Kafka and Pandas (ii) More check types e.g. compare datasets when releasing new analytics code (iii) Natively support additional orchestration and CI/CD tools

Would love to hear your thoughts!

Regarding YAML/DSL being a trap. Did you consider a fallback option to allow users to just plug in SQL? See this for example: https://docs.soda.io/soda-cl/user-defined.html