Re:Querying Parquet with Precision Using DuckDB
by mkarlsch on Friday June 25, 2021 @05:31PM
(#27636525)
We used Apache Drill a few years back and it could do exactly that - run locally and provide a SQL interface to query parquet files. Still great to see more and better options!
[ Parent
| on HN ]
Re:Querying Parquet with Precision Using DuckDB
by wenc on Friday June 25, 2021 @05:46PM
(#27636670)
Drill is a fair choice (it supports tons of formats), but for me it was a bit heavy -- it required Java and plus it returned results in a form that wasn't easily manipulated (the result is a table in the console drawn with text-chars [1]). If you had a really long result set, you couldn't do a buffered scroll due to text UI limitations -- everything gets printed out and you have to either do manual pagination using LIMIT clauses or just have a really long scrollback buffer. This kind of breaks the continuity of many data analysis workflows.
With DuckDB, the return result is a Python data structure (list of tuples), which is amenable to further manipulation without exporting to another format.
Many GUI db tools often return results in a tabular form which can be copy-pasted into other GUI apps (or copy-pasted as CSV).
[1] https://drill.apache.org/docs/drill-in-10-minutes/#querying-...
Re:Querying Parquet with Precision Using DuckDB
by sixdimensional on Saturday June 26, 2021 @12:15AM
(#27639055)
You didn't connect to drill using JDBC? You can use any open source GUI with that, like DBeaver.