Re:Relational Operators in Apache Calcite
by mickeyp on Tuesday April 11, 2023 @02:05AM
(#35521409)
It is and it isn't. There's some good stuff there, but there's ever the risk that you'll get bushwhacked by something that you need to pay an expensive consultancy -- who just so happens to be the team behind the product -- to fix.
Though I feel bad for the person who decides to peruse Apache's list of projects and decide what technology to use based solely on the high-level overviews. Good luck picking the wheat from the chaff!
Re:Relational Operators in Apache Calcite
by michaelmior on Tuesday April 11, 2023 @07:14AM
(#35523458)
I'll just note that the risk mentioned certainly does not apply to Apache Calcite. (Disclaimer: I am a member of the Apache Calcite PMC, but not terribly active at the moment.)
Re:Relational Operators in Apache Calcite
by di4na on Tuesday April 11, 2023 @02:19AM
(#35521477)
Yes and no.
A lot of projects in Apache control started as one (big) company project, got open sourced then move to Apache, in order to allow collaboration with a "community" without as much conflict of interest.
But then the original sponsor also usually lose interest and/or no community get creates. Hence, graveyard.
Or it got opensource as last ditch try to keep it running after the sponsor stopped wanting it. And then it is usually a graveyard move.
It is not that Apache kills them. It is that they were already dead and Apache is the place that take them in to accompany their demise.
Re:Relational Operators in Apache Calcite
by jamesblonde on Tuesday April 11, 2023 @02:07AM
(#35521417)
A lot of open-source Apache projects were backed by commercial companies, and clouds (primarly AWS) forked those projects competing directly with the companies backing the Apache projects. The end result was that the commercial companies mostly released new versions of the software with more restrictive licenses that could not be sold for free by the cloud companies.
That doesn't make Apache software a graveyard. It just makes that go-to-market open-source model leveraging Apache's brand a dead-end.
Re:Relational Operators in Apache Calcite
by kitd on Tuesday April 11, 2023 @06:55AM
(#35523334)
It depends. There are a large number of dead projects. There are also a large number of libraries and tools that are active and widely depended on, but you may not have heard of them.
Then there are quite a few large projects donated by corps with active communities. See this page
Re:Relational Operators in Apache Calcite
by michaelmior on Tuesday April 11, 2023 @07:08AM
(#35523417)
It's certainly not the case for Apache Calcite. Calcite is pretty widely used by a number of companies and there are a number of highly active developers.
Re:Relational Operators in Apache Calcite
by BenoitP on Tuesday April 11, 2023 @03:45PM
(#35530448)
Calcite is relatively unknown, but it is the base so many other projects are built on. Among them is Apache Flink, which is quite alive and top of its category (low latency distributed stream processing)
Re:Relational Operators in Apache Calcite
by noisy_boy on Tuesday April 11, 2023 @02:21AM
(#35521498)
I recently had to parse source code of views to produce a mapping between the view columns and the underlying table columns - it basically worked with some caveats and was pretty painful to do (most databases expose this as information schema/metadata but that was not an option for us for various reasons).
For anyone who is knowledgable about Calcite features, is this something that Calcite can produce without having to write a ton of code (which is what my current solution is)? I would find it very satisfying to replace hundreds of lines of bespoke code with a few library calls.
Re:Relational Operators in Apache Calcite
by michaelmior on Tuesday April 11, 2023 @07:12AM
(#35523446)
This should be pretty easily doable in Calcite if you have access to the query which defines the view. Calcite has a pretty rich metadata system. Take a look at RelMetadataQuery.getColumnOrigin.
Re:Relational Operators in Apache Calcite
by hobofan on Tuesday April 11, 2023 @03:02AM
(#35521785)
I'd recommend looking into Substrait[0] for anyone interested in this mid-/low-level database/query engine space. It tries to establish a vendor-agnostic set of relational operators (and all the things surrounding it) to make it easier for query engine implementations to share individual components.
Re:Relational Operators in Apache Calcite
by yeesian on Tuesday April 11, 2023 @09:33PM
(#35533979)
That's cool, thanks for sharing! Do you know how close they are to their example use cases [1]? So far I've only been able to find a tool for calcite SQL parsing [2] but not the portion connecting to Arrow C++ compute kernel yet.
Re:Relational Operators in Apache Calcite
by hobofan on Wednesday April 12, 2023 @01:20AM
(#35535326)
I'd check out the Slack, which is where I've seen a few projects integrating it coordinating.
I think DuckDB is one of the projects that has the best support for executing Substrait query plans. I think for most other projects there are some forks with substrait support (e.g. datafusion) but nothing merged to upstream yet.
I'm not sure if there are any systems where it is integrated and yields tangible benefits yet (though there is decent progress on common tooling, so that shouldn't be too far in the future).
Is that really the case?
Though I feel bad for the person who decides to peruse Apache's list of projects and decide what technology to use based solely on the high-level overviews. Good luck picking the wheat from the chaff!
A lot of projects in Apache control started as one (big) company project, got open sourced then move to Apache, in order to allow collaboration with a "community" without as much conflict of interest.
But then the original sponsor also usually lose interest and/or no community get creates. Hence, graveyard.
Or it got opensource as last ditch try to keep it running after the sponsor stopped wanting it. And then it is usually a graveyard move.
It is not that Apache kills them. It is that they were already dead and Apache is the place that take them in to accompany their demise.
That doesn't make Apache software a graveyard. It just makes that go-to-market open-source model leveraging Apache's brand a dead-end.
Then there are quite a few large projects donated by corps with active communities. See this page
https://projects.apache.org/statistics.html
and scroll down to "Largest/Busiest repositories". There's a list of widely used enterprise software in there which are definitely not dead.
https://calcite.apache.org/docs/powered_by.html
For anyone who is knowledgable about Calcite features, is this something that Calcite can produce without having to write a ton of code (which is what my current solution is)? I would find it very satisfying to replace hundreds of lines of bespoke code with a few library calls.
[0]: https://substrait.io
[1]: https://substrait.io/#example-use-cases [2]: https://substrait.io/tools/producer_tools/
I think DuckDB is one of the projects that has the best support for executing Substrait query plans. I think for most other projects there are some forks with substrait support (e.g. datafusion) but nothing merged to upstream yet.
I'm not sure if there are any systems where it is integrated and yields tangible benefits yet (though there is decent progress on common tooling, so that shouldn't be too far in the future).