That's a fair question, and I guess my answer would be "ergonomics". I tend to work in Python not Scala, and there's a certain amount of boilerplate that is necessary to even get Spark functional. For instance, this is a code snippet to spin up local Spark (not Spark Shell) I have stored in a file which I have to look up because there's no way I can commit this to memory:

    import findspark
    findspark.init()
    from pyspark.sql import SparkSession

    sqlContext = SparkSession.builder.master("local[*]").getOrCreate()
    df = sqlContext.read.parquet('path-to-file/commentClusters.parquet')

Not to mention fussy things you have to do in Spark to alias a table:

    df.createOrReplaceTempView("tableA")
    sqlContext.sql("SELECT count(*) from tableA").show()

Compare this to these 2 (!) DuckDB commands in the Python REPL that I just ran moments ago:

    >>> import duckdb
    >>> duckdb.query("SELECT COUNT(*) FROM '*.parquet'").fetchall()
    [(2,)]

> I tend to work in Python not Scala

Fair enough. I’m more comfortable in Scala than Python. Have you tried using a local Jupyter notebook? That would seem to tick all the boxes. https://jupyter.org/install

I do use Jupyter and the boilerplate code above is what I usually deploy in Jupyter.

Spark syntax is still just a touch more complicated than necessary for my needs because it's solving the much more general distributed computing problem. (not to mention the impedance mismatch -- Spark is very much a Scala native app, and it feels that way to a Python user).

DuckDB on the other hand seems to provide a quick way to interact with local Parquet files in Python, which I appreciate.