Directly running DuckDB queries on data stored in SQLite files

Re:Directly running DuckDB queries on data stored in SQLite ... by wenc on Friday March 25, 2022 @06:17PM (#30807541)

Exciting stuff!

I’ve been using duckdb instead of Pandas these days because it’s much faster on larger datasets plus I can write more compact and complex SQL than I can Pandas constructs. The fact that duckdb can query in memory Pandas dataframes faster than Pandas itself is a plus.

It’s like having a local performant database engine that can query and join across Parquet, CSV, Pandas and now SQLite.

Kinda like a quick efficient local Spark.

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by tanin on Friday March 25, 2022 @10:21PM (#30808745)

I've built an offline desktop for this: https://superintendent.app

It is a wrapper on Sqlite, but I'm looking to switch to duckdb because its dialect is more comprehensive.

Previously, I was using ruby, sometimes python, and sometimes postgresql to process CSV. But it wasn't convenient enough. Most of the times I just tried to use Excel, but formula is so hard to use.

[ Parent | on HN ]