Re:Show HN: WarcDB: Web crawl data as SQLite databases
by marginalia_nu on Sunday June 19, 2022 @01:33PM
(#31801465)
Seems like the benefit of sqlite is the sort of usecases where you maybe don't want to load everything you've crawled into a search engine, but want to be able to cherrypick the data and retrieve specific documents for further processing. Which is certainly a use case that exists, and indeed not really what WARC is designed for.
[ Parent
| on HN ]
Re:Show HN: WarcDB: Web crawl data as SQLite databases
by traverseda on Sunday June 19, 2022 @01:52PM
(#31801640)
Great for most end user facing applications though.
Re:Show HN: WarcDB: Web crawl data as SQLite databases
by marginalia_nu on Sunday June 19, 2022 @02:00PM
(#31801701)
End-user facing applications usually don't consume website crawls, do they? That's impractical for many reasons, the sheer size alone being perhaps the biggest obstacle.
If you want to do something like have an offline copy of a website, ZIM[1] is a far more suitable format as it's extremely space-efficient and also fast.