Show HN: WarcDB: Web crawl data as SQLite databases

Re:Show HN: WarcDB: Web crawl data as SQLite databases by nlohmann on Sunday June 19, 2022 @01:36PM (#31801491)

Have you every played with SQLite virtual tables (https://sqlite.org/vtab.html) - they could allow to provide an SQLite interface while keeping the same structure on disk. Though it requires a bit of work (implementing the interface can be tedious), it can avoid the conversion in the first place.

[ Parent | on HN ]

Re:Show HN: WarcDB: Web crawl data as SQLite databases by fforflo on Sunday June 19, 2022 @02:00PM (#31801705)

Good point. Actually CommonCrawl provides Parquet files for their archives too.

And there's this vtable for Parquet extension. https://github.com/cldellow/sqlite-parquet-vtable

But for my use case virtual would be too complicated.

[ Parent | on HN ]

Re:Show HN: WarcDB: Web crawl data as SQLite databases by m_ke on Sunday June 19, 2022 @10:40PM (#31805743)

DuckDB would probably be a way better option and works amazingly well on top of parquet (https://duckdb.org/docs/data/parquet)

[ Parent | on HN ]

Re:Show HN: WarcDB: Web crawl data as SQLite databases by fforflo on Sunday June 19, 2022 @04:11PM (#31802964)

Then again, do you need virtual tables? The .warc structure won't change, so the tables won't change. But you can have SQL views defined instead for common queries.

[ Parent | on HN ]