Re:Launch HN: Hydra (YC W22) – Query any database via Postgres
by michaelmior on Thursday February 24, 2022 @06:42AM
(#30453627)
I was moreso referring to the pieces of queries that Calcite may have to execute itself, often because they are not supported by the underlying data source. That piece of Calcite hasn't been very highly optimized. There's definitely ongoing work in that area though.
For example, I have a former student who's been working on an adapter to Apache Arrow. Beyond having a fast execution engine, this should also eventually enable Calcite to work pretty seamlessly with other systems using Arrow without serialization overhead.
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres
by gavinray on Thursday February 24, 2022 @08:36AM
(#30454467)
Is this student in contact with Jacques Nadeau?
He's one of the only people I know that's a committer/PMC of both Arrow and Calcite, super nice and brilliant guy.
Maybe he'd have good feedback or ideas around this?
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres
by michaelmior on Thursday February 24, 2022 @08:45AM
(#30454560)
Indirectly. I know Jacques. I'm also on the Calcite PMC :) Thanks for the pointer though!
I am using it as the backbone for my hobby project that auto-generated federated GraphQL API's:
https://github.com/GavinRay97/GraphQLCalcite
The experience has been incredibly positive and the community has been incredibly helpful & supportive. It's one of the coolest technical projects I've ever seen and has sparked my interest in query engines and relational databases.
I posted some JMH benchmarks of an app that parses a GraphQL query, converts it to Calcite relational expressions, and then executes it against an in-memory DB and it ran on the orders of milliseconds:
https://lists.apache.org/thread/hofjx628864t0kt4kk8vo4tjfrxb...
Something very similar to Calcite but much lesser-known is the "Teiid" project:
https://github.com/teiid/teiid
Highly recommend checking the code out. It's got a brilliant query optimizer/planner tailored to cross-datasource queries, a cache system, and a translator architecture that can convert a generic SQL dialect to dozens of SQL/NoSQL flavors.
Also integrates other data sources like REST API's, S3, flat files etc as queryable data sources.