News for hackers. Stuff that matters.
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by michaelmior on Wednesday February 23, 2022 @12:41PM (#30443863)
For anyone interested, Apache Calcite[0] is an open source data management framework which seems to do many of the same things that Hydra claims to do, but taking a different approach. Operating as a Java library, Calcite contains "adapters" to many different data sources from existing JDBC connectors to Elasticsearch to Cassandra. All of these different data sources can be joined together as desired. Calcite also has it's own optimizer which is able to push down relevant parts of the query to the different data sources. However, you get full SQL on data sources which don't support it, with Calcite executing the remaining bits itself.

Generally all that is required to connect to multiple data sources from CSV to Elasticsearch is just writing a JSON configuration file. Then can get SQL access via JDBC with the able to join all those sources together.

Unfortunately, I would not be too surprised if the query execution Calcite was found to be less performance-optimized than Hydra. There is ongoing work for improvement there. That said, there are users of Calcite at Google, Uber, Spotify, and others who have made great use of various parts of the framework.

[0] https://calcite.apache.org/

Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by gavinray on Wednesday February 23, 2022 @05:39PM (#30447580)
Calcite is actually pretty damn fast, the overhead is surprisingly minimal.

I am using it as the backbone for my hobby project that auto-generated federated GraphQL API's:

https://github.com/GavinRay97/GraphQLCalcite

The experience has been incredibly positive and the community has been incredibly helpful & supportive. It's one of the coolest technical projects I've ever seen and has sparked my interest in query engines and relational databases.

I posted some JMH benchmarks of an app that parses a GraphQL query, converts it to Calcite relational expressions, and then executes it against an in-memory DB and it ran on the orders of milliseconds:

https://lists.apache.org/thread/hofjx628864t0kt4kk8vo4tjfrxb...

Something very similar to Calcite but much lesser-known is the "Teiid" project:

https://github.com/teiid/teiid

Highly recommend checking the code out. It's got a brilliant query optimizer/planner tailored to cross-datasource queries, a cache system, and a translator architecture that can convert a generic SQL dialect to dozens of SQL/NoSQL flavors.

Also integrates other data sources like REST API's, S3, flat files etc as queryable data sources.

Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by michaelmior on Thursday February 24, 2022 @06:42AM (#30453627)
I was moreso referring to the pieces of queries that Calcite may have to execute itself, often because they are not supported by the underlying data source. That piece of Calcite hasn't been very highly optimized. There's definitely ongoing work in that area though.

For example, I have a former student who's been working on an adapter to Apache Arrow. Beyond having a fast execution engine, this should also eventually enable Calcite to work pretty seamlessly with other systems using Arrow without serialization overhead.

Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by gavinray on Thursday February 24, 2022 @08:36AM (#30454467)
Is this student in contact with Jacques Nadeau?

He's one of the only people I know that's a committer/PMC of both Arrow and Calcite, super nice and brilliant guy.

Maybe he'd have good feedback or ideas around this?

https://twitter.com/intjesus

Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by michaelmior on Thursday February 24, 2022 @08:45AM (#30454560)
Indirectly. I know Jacques. I'm also on the Calcite PMC :) Thanks for the pointer though!
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by gavinray on Thursday February 24, 2022 @09:06AM (#30454802)
If you've responded to any of my emails, no thank YOU!
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by sdesol on Wednesday February 23, 2022 @03:22PM (#30445895)
I've never heard of Calcite before and dug a bit deeper into this project and found it to be quite active. Only taking into consideration changes to Java files, this is the repo activity for the last 12 months:

     month  | authors | commits | files | churn 
   ---------+---------+---------+-------+-------
    2022-02 |      21 |      62 |    84 |  4387
    2022-01 |      26 |     109 |   243 | 32429
    2021-12 |      33 |      99 |   198 | 10461
    2021-11 |      19 |      49 |    77 |  6960
    2021-10 |      26 |      64 |   371 | 13626
    2021-09 |      18 |      41 |    68 |  2258
    2021-08 |      11 |      17 |    25 |  1924
    2021-07 |      17 |      30 |    51 |  2704
    2021-06 |      14 |      31 |    28 |  1708
    2021-05 |       9 |      17 |    35 |  1606     
    2021-04 |      11 |      46 |    99 |  4224
    2021-03 |      16 |      36 |   143 |  8471
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by michaelmior on Thursday February 24, 2022 @06:38AM (#30453597)
The community is also great. Honestly, the main thing preventing things from moving faster is lack of time to review PRs. There's a lot of interesting things in the pipe!
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by sdesol on Thursday February 24, 2022 @11:05AM (#30456297)
Yeah I can see what you mean. There are currently 20 open pull requests [1] that hasn't changed in over 28 days, which accounts for 53% of all open pulls that are less than 4 months old.

1) https://oss.gitsense.com/insights/github?q=pull-age%3A%3C%3D...

Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by riverdroid on Wednesday February 23, 2022 @04:53PM (#30447040)
What do you use to produce this analysis?
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by sdesol on Wednesday February 23, 2022 @05:39PM (#30447578)
My product (https://gitsense.com) moves most of Git's history into a Postgres database and from there, you can execute the following SQL statement:

    select
        commit_ym AS month,
        count(distinct(author_email)) as authors,
        count(distinct(commit_id)) as commits,
        count(distinct(path_id)) as files,
        sum(total) as churn
    from
        z1_commits_422 as commits,
        z1_changes_422 as changes,
        z1_code_churn_422 as churn
    where
        commits.id=changes.commit_id and
        changes.code_churn_id=churn.id and
        lang='java'
    group by commit_ym
    order by commit_ym desc
    limit 12
By having most of Git's history in SQL, I can slice, dice and cross-reference code history, which is how my product works.
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by wuputah on Wednesday February 23, 2022 @02:34PM (#30445270)
Calcite is definitely some cool tech. I can see why it would be attractive for bigger teams, but it seems like a big lift for smaller teams. Our goal is to make it easy for devs already familiar with Postgres to be able to use add databases without learning new tools or adding software... besides adding Hydra, of course!
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by michaelmior on Wednesday February 23, 2022 @02:51PM (#30445499)
That's a fair point. Although if you're already using a JVM language, it's incredibly easy to integrate. Just another JDBC data source with some JARs to add to your classpath :)
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by imachine1980_ on Wednesday February 23, 2022 @01:40PM (#30444640)
What's the difference whit regular ORM?(genuine ask)
Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by michaelmior on Wednesday February 23, 2022 @02:31PM (#30445229)
ORMs map relational databases to objects. Calcite does not do that. Calcite takes different data sources (potentially with different data models) and presents them all in the relational model. You interact with your data entirely in SQL while ORMs typically have some DSL which uses whatever object model the ORM defines. ORMs are also typically designed to connect to a single data source at a time. I'm personally not familiar with any ORMs that allow combining data from multiple sources within the same application.

The primary advantage of Calcite for connecting multiple data sources are 1) easily joining data sources that use completely different APIs (assuming there is a Calcite adapter available) and 2) supporting more complex queries than the original data source supports without having to write code (other than SQL) to do the processing.

Re:Launch HN: Hydra (YC W22) – Query any database via Postgres by gavinray on Wednesday February 23, 2022 @05:48PM (#30447665)
Don't forget having two industrial-grade academic query planners/optimizer implementations that took collective decades of engineering effort!

And the project being founded by someone who has written multiple RDBMS.

Calcite is a wonder.