News for hackers. Stuff that matters.

A comment on Ask HN: I want to crawl every plain HTML website. Where do I begin?.

Re:Ask HN: I want to crawl every plain HTML website. Where d... by ratio11 on Friday December 02, 2022 @02:55AM (#33827998)
You might be interested in Common Crawl. They crawl the internet and make the full dataset downloadable.

https://commoncrawl.org

[ Parent | on HN ]

There are no comments on this comment yet.