Speaking for an Engineer unfortunately often data is simple CSV. Parsing this data robustly and visualizing it without scripting is still something that hasn't been solved.

If you have any idea how to do it, I'd be more than glad to hear about it.

You can visualize CSVs easily without scripting: use Tableau or some other BI tool [1] (not free though).

There are free alternatives, but many require programming.

[1] https://www.absentdata.com/tableau-alternatives/

Parquet is a great columnar format, and Avro is great for schema evolution. But they're not that easy to work with directly. To this day there isn't a decent Parquet viewer to do adhoc data viewing.

SQLite on the other hand has a lightweight SQL REPL that can be invoked from the command line.

Spark can work with SQLite via JDBC, though obviously it isn't as native as Parquet. Between SQLite and Parquet, I might pick Parquet under most circumstances.

But it seems to me SQLite ought to at least be a better option than CSV for Spark jobs (less work needed to do type inference, predicate pushdowns are trivial, etc.)

csv is notoriously bad with spark - in fact you couldn't write to csv for the longest time without external libraries like the databricks one, so I don't contest your last point.

SQlite is usually a single file representing a database though, I don't know how it would work with partitioning and stuff, and then how to handle the schema evolving across sqlite files.

Also, the now native Spaek CSV export doesn’t follow RFC 4180 with default settings, it escapes double quotes with a backlash. So odd.