Some more notes that might be of use:
* We also have a command line interface (with syntax highlighting) [1]
* You can use .df() instead of .fetchall() to retrieve the results as a Pandas DataFrame. This is faster for larger result sets.
* You can also use DuckDB to run SQL on Pandas DataFrames directly [2]
I would poke at local Parquet files either through a local instance of Spark (or DataBricks when I was using that for remote data folders) or use Pandas' read_parquet from Jupyter.
With DuckDB, the return result is a Python data structure (list of tuples), which is amenable to further manipulation without exporting to another format.
Many GUI db tools often return results in a tabular form which can be copy-pasted into other GUI apps (or copy-pasted as CSV).
[1] https://drill.apache.org/docs/drill-in-10-minutes/#querying-...
What's the burning pain-point here? Spinning up a spark shell is extremely quick and easy.
import findspark
findspark.init()
from pyspark.sql import SparkSession
sqlContext = SparkSession.builder.master("local[*]").getOrCreate()
df = sqlContext.read.parquet('path-to-file/commentClusters.parquet')
df.createOrReplaceTempView("tableA")
sqlContext.sql("SELECT count(*) from tableA").show()
>>> import duckdb
>>> duckdb.query("SELECT COUNT(*) FROM '*.parquet'").fetchall()
[(2,)]Fair enough. I’m more comfortable in Scala than Python. Have you tried using a local Jupyter notebook? That would seem to tick all the boxes. https://jupyter.org/install
Spark syntax is still just a touch more complicated than necessary for my needs because it's solving the much more general distributed computing problem. (not to mention the impedance mismatch -- Spark is very much a Scala native app, and it feels that way to a Python user).
DuckDB on the other hand seems to provide a quick way to interact with local Parquet files in Python, which I appreciate.
I've always felt that Parquet (or ORC) ought to be more popular mainstream formats than they are, but in practice the interactive tooling for Parquet has always been weak. For years, there wasn't a good GUI viewer (well, there were a few half-hearted attempts) or a good SQL query tool for Parquet like there is for most relational databases, and lazily examining the contents of a large Parquet folder from the command line has always been a chore. At least now, DuckDB's parquet support moves the needle on the last point.