News for hackers. Stuff that matters.

A comment on Launch HN: Elementary (YC W22) – Open-source data observability. Parent comment

Re:Launch HN: Elementary (YC W22) – Open-source data observa... by Maayansa on Friday March 04, 2022 @04:45PM (#30561032)
While writing our dbt project we looked into more than 60 dbt projects to learn from prior work while developing Elementary, and have been inspired by different things in different places. You're right that we were inspired by a couple of techniques you used, one being that creative way to improve performance (though the 4 thread setting itself is the dbt recommendation in their docs). Another is using z-score for anomaly detection, which we saw in a number of related projects and it's widely used in the industry.

In terms of the lineage, you can see in the code that we mostly rely on query and access history that exist in Snowflake and Bigquery to parse the queries and learn about the connection between nodes in the graph. We use other python libraries like sqlfluff and sqllineage as low level parsers for some specific use cases which we extend and solve many things on top of them. Actually we're heavy open source users, depending on around 20 libraries, all MIT or Apache.

[ Parent | on HN ]
Re:Launch HN: Elementary (YC W22) – Open-source data observa... by mateuszklimek on Sunday March 06, 2022 @04:18AM (#30576202)
Okay, I'm happy that you admit inspiration in this comment (in opposition to the previously deleted one).

Also, I think it's more than just following up re_data in a couple of places. Elementary's whole data monitoring part started much later than your Lineage part, and it seems to try to follow what re_data did there on the idea & implementation level. I'm sure the other 59 projects you mention were not dbt packages for data reliability (there were no other one in the dbt hub) which is what re_data is and now elementary also tries to copy this. (seeing our traction)

As mentioned, it's open-source. You can use our code. But if you are doing that, state that clearly in the LICENSE.

[ Parent | on HN ]
Re:Launch HN: Elementary (YC W22) – Open-source data observa... by nuclearnice1 on Friday March 04, 2022 @05:34PM (#30561522)
I think mateuszklimek is pointing out that the MIT license requires you to include the redata copyright in your source.
[ Parent | on HN ]
Re:Launch HN: Elementary (YC W22) – Open-source data observa... by DarthNebo on Saturday March 05, 2022 @06:07AM (#30566390)
Right on point, they don't even have a filled out LICENSE on the repo

> Copyright [yyyy] [name of copyright owner]

https://github.com/elementary-data/elementary/blob/master/LI...

[ Parent | on HN ]