News for hackers. Stuff that matters.

A comment on Show HN: WarcDB: Web crawl data as SQLite databases. Parent comment

Re:Show HN: WarcDB: Web crawl data as SQLite databases by fforflo on Sunday June 19, 2022 @02:00PM (#31801705)
Good point. Actually CommonCrawl provides Parquet files for their archives too.

And there's this vtable for Parquet extension. https://github.com/cldellow/sqlite-parquet-vtable

But for my use case virtual would be too complicated.

[ Parent | on HN ]
Re:Show HN: WarcDB: Web crawl data as SQLite databases by m_ke on Sunday June 19, 2022 @10:40PM (#31805743)
DuckDB would probably be a way better option and works amazingly well on top of parquet (https://duckdb.org/docs/data/parquet)
[ Parent | on HN ]