News for hackers. Stuff that matters.

A comment on Fast analysis with DuckDB and Pyarrow. Parent comment

Re:Fast analysis with DuckDB and Pyarrow by 41b696ef1113 on Saturday April 30, 2022 @08:50PM (#31221007)
I would say most of the tooling around Parquet sucks. A lot of simple use cases (head, sample, merge, etc) are locked behind rather heavyweight ecosystems (Hadoop, Spark, or other JVM dependencies). Newer tools are starting to develop that make the situation better.

Is Parquet better/faster/stronger than keeping everything in schemaless CSVs? 100%, but it has historically meant that I have to make trade-offs to benefit the tooling rather than how I want to interactively approach my data.

Most people do not have "big data" problems where the performance differences of Parquet vs csv matter.

[ Parent | on HN ]
Re:Fast analysis with DuckDB and Pyarrow by vanpythonista on Sunday May 01, 2022 @01:59AM (#31222589)
Shameless plug, my project exposes the common Parquet operations using a Rust CLI tool that utilizes the Rust API for Apache Arrow and can be used without any Java/Hadoop/Spark dependencies. Also available a static binary.

https://github.com/manojkarthick/pqrs

[ Parent | on HN ]
Re:Fast analysis with DuckDB and Pyarrow by 41b696ef1113 on Sunday May 01, 2022 @03:12AM (#31222867)
That looks great! Exactly the kind of tooling required to allow me to feel more comfortable locking up human readable data into a blob. Being a single Rust binary is just the cherry on top.

I might submit a few feature requests, but one that immediately comes to mind: csv -> parquet. Perhaps out of scope for the original vision, but having a single utility that could roundtrip data would be fantastically useful.

[ Parent | on HN ]
Re:Fast analysis with DuckDB and Pyarrow by vanpythonista on Sunday May 01, 2022 @07:34AM (#31224153)
Glad you like it. A CSV->Parquet conversion command would certainly be really useful!
[ Parent | on HN ]
Re:Fast analysis with DuckDB and Pyarrow by 1egg0myegg0 on Sunday May 01, 2022 @08:53AM (#31224632)
I hope that DuckDB can help here! You can use the CLI or Python client to read a parquet file, but my favorite for quick exploration is popping open DBeaver (a SQL IDE) and using the DuckDB JDBC driver. That way you can see some column metadata, and even open multiple files, etc. I wrote a short guide for that here: https://duckdb.org/docs/guides/sql_editors/dbeaver
[ Parent | on HN ]