Directly running DuckDB queries on data stored in SQLite files

Re:Directly running DuckDB queries on data stored in SQLite ... by mastax on Friday March 25, 2022 @07:03PM (#30807829)

I really like the DuckDB philosophy, and the maintainers were really responsive and helpful with my docs questions. Didn't end up using it because the compressed database files were much larger than SQLite for my data (weird use case, admittedly), and the .NET libraries were poor. I look forward to trying it out again, though.

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by wenc on Friday March 25, 2022 @09:02PM (#30808411)

> the compressed database files were much larger than SQLite for my data

Just curious, what format did you use?

If it's Parquet, it does column compression via Snappy by default (which more optimized for speed than size).

You can change the compression to ZSTD or GZIP for higher compression ratios.

https://duckdb.org/docs/data/parquet

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by mastax on Friday March 25, 2022 @11:25PM (#30809005)

I tried all of the available compression formats in parquet, natively in duckdb, and compressing raw duckdb files as a whole. IIRC just GZIPing SQLite files beat them all. The SQLite files were much smaller uncompressed (~220mb vs 800mb) which was probably most of the difference.

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by pletnes on Saturday March 26, 2022 @07:39AM (#30810993)

Can you still query a gzipped sqlite file? What’s the sense in gzipping it if not?

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by floydnoel on Saturday March 26, 2022 @01:02PM (#30813462)

Do other formats (Snappy?) allow this?

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by wenc on Wednesday March 30, 2022 @10:39AM (#30855821)

You can also query gzip and zstd parquet files directly.

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by benglish11 on Saturday March 26, 2022 @02:43PM (#30814206)

You can query snappy parquet files directly.

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by ignoramous on Saturday March 26, 2022 @03:27AM (#30809986)

> ...the .NET libraries were poor.

DuckDB's Node bindings leave a lot to be desired, as well. WASM bindings have a different API surface, too. From my reading of the code and docs, it looks like duckdb 3p bindings for anything other than python is in dire need fit and polish.

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by ankoh on Saturday March 26, 2022 @01:32PM (#30813665)

We're still on a journey to explore what APIs work best with JavaScript but the differences between WASM and Node are on purpose. DuckDB-Wasm has isolated wasm heap memory and runs in a separate web worker. That's why we serialize everything as Arrow IPC buffer and pass the ArrayBuffer through the workers message API as transferrable. On Node.js, we can interact with DuckDB much more easily and don't want to pay the price for the IPC stream every time. The truth is, we tried quite a few different APIs and this turned out the be a good trade off between efficiency and convenience.

But I'm happy to discuss this further if you have any suggestions.

[ Parent | on HN ]