Re:Ask HN: Best way to keep the raw HTML of scraped pages?
by crazygringo on Friday November 11, 2022 @03:28PM
(#33566356)
Yup. A database gives you all the performance AND flexibility you need. MySQL or PostgreSQL will work well too.
Storing pages as files is a no-go because it wastes way too much disk space due to block sizes. While more customized cache tools will never be as flexible or have as much tooling as a widely supported relational database.
For even better compression use a preset dictionary as well tuned to a wide sample of HTML, but it doesn't sound like you need to go that far.
Storing pages as files is a no-go because it wastes way too much disk space due to block sizes. While more customized cache tools will never be as flexible or have as much tooling as a widely supported relational database.
For even better compression use a preset dictionary as well tuned to a wide sample of HTML, but it doesn't sound like you need to go that far.