Directly running DuckDB queries on data stored in SQLite files

Re:Directly running DuckDB queries on data stored in SQLite ... by wenc on Friday March 25, 2022 @09:02PM (#30808411)

> the compressed database files were much larger than SQLite for my data

Just curious, what format did you use?

If it's Parquet, it does column compression via Snappy by default (which more optimized for speed than size).

You can change the compression to ZSTD or GZIP for higher compression ratios.

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by mastax on Friday March 25, 2022 @11:25PM (#30809005)

I tried all of the available compression formats in parquet, natively in duckdb, and compressing raw duckdb files as a whole. IIRC just GZIPing SQLite files beat them all. The SQLite files were much smaller uncompressed (~220mb vs 800mb) which was probably most of the difference.

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by pletnes on Saturday March 26, 2022 @07:39AM (#30810993)

Can you still query a gzipped sqlite file? What’s the sense in gzipping it if not?

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by floydnoel on Saturday March 26, 2022 @01:02PM (#30813462)

Do other formats (Snappy?) allow this?

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by wenc on Wednesday March 30, 2022 @10:39AM (#30855821)

You can also query gzip and zstd parquet files directly.

[ Parent | on HN ]

Re:Directly running DuckDB queries on data stored in SQLite ... by benglish11 on Saturday March 26, 2022 @02:43PM (#30814206)

You can query snappy parquet files directly.

[ Parent | on HN ]