News for hackers. Stuff that matters.
Re:Do we need a third Apache project for columnar data repre... by sixdimensional on Tuesday October 31, 2017 @04:25PM (#15596247)
Calcite is more a query optimization framework. Actually, I think parent has a good question, which is, how to use these tools, formats, etc. from something simpler than an entire full blown distributed technology - and I think the answer to that is, there aren't too many technologies out of the box that do that right now.

I would think maybe extending SQLite to use columnar storage would be one possible path that might make sense.

There are columnar DBMS engines you can embed like MonetDB, but I personally haven't seen a lightweight query engine sitting right on top of Parquet/ORC, or implementing Arrow for in-memory columnar. You can get Apache Spark up and running pretty quickly on a single node, but I haven't seen anything even easier than that.

There are no comments on this comment yet.