Maayansa

Hi, thank you! The way we see it, AWS Deeque, as well as Great Expectations and dbt tests, are used for static data testing. This is great for many use cases, however there are problems you will only detect by continuously monitoring. Just like in software engineering you use both unit testing and monitoring.

While writing our dbt project we looked into more than 60 dbt projects to learn from prior work while developing Elementary, and have been inspired by different things in different places. You're right that we were inspired by a couple of techniques you used, one being that creative way to improve performance (though the 4 thread setting itself is the dbt recommendation in their docs). Another is using z-score for anomaly detection, which we saw in a number of related projects and it's widely used in the industry.

In terms of the lineage, you can see in the code that we mostly rely on query and access history that exist in Snowflake and Bigquery to parse the queries and learn about the connection between nodes in the graph. We use other python libraries like sqlfluff and sqllineage as low level parsers for some specific use cases which we extend and solve many things on top of them. Actually we're heavy open source users, depending on around 20 libraries, all MIT or Apache.

Thank you! We believe that this upload is super valuable and could unlock a lot of additional use cases. We are already working on some of these and will release in the next few weeks.

Hi, it's defiantly in our plans for the next few weeks. As I mentioned in the post we leverage dbt for the whole data monitoring layer. We wrote the package with all the cross-db best practices, but didn't test it on Redshift so there are probably some gaps. There are a few users on our community on Slack that are Redshift users and they will help us to test on production with real data. Hopefully it will not require much to make it run smoothly.

Thank you, would love to get your feedback and thoughts on what we should add.