Does OctoSQL support reading columnar compressed formats (ex. Parquet) from distributed storage (ex. S3) ?
No, we don't support Excel, JSON, CSV and Parquet datasources other than local files yet, though that's definitely planned and would be very easy to add.
Drill does ;-)
Drill also has a streaming Excel reader that works very well with large Excel files...
Re:Apache Drill's Future
by willvarfar on Wednesday May 20, 2020 @04:49PM
(#23251622)
How does it compare to prestosql?
Much less mature.
There's no distributed execution yet.
Easy to get going locally.
I'm not sure but I think Presto is in-memory? We're optimizing for SSD disks to easily support big states, and achieve durability this way. SSD disks are still plenty fast.
Definitely better streaming support. We're very much concentrating on good streaming ergonomics. OctoSQL isn't a batch execution engine at its heart, it's a streaming one.
If you're interested in Drill, check out OctoSQL[0]. It shares the same vision of querying multiple datasources using pure SQL, and pushing down as much operations as possible to the underlying datasource.
Moreover, there's been a huge rewrite under way this past year, ready to use on the master branch, yet unreleased however (will be available soon).
It adds Kafka and Parquet support and most importantly first-class unbounded stream support. Including temporal SQL extensions for working with event time metadata (instead of system time) so you can use stuff like live updated time window aggregations on incoming kafka streams. It also now uses on-disk badger storage as the primary way to store its state, so you can do Group Bys / Joins with lots of keys, and restarts of OctoSQL won't alter the final result (exactly-once semantics).
Make sure to check it out, it's also very simple to get going locally!
Disclosure: I'm one of the main contributors.
[0]:https://github.com/cube2222/octosql