News for hackers. Stuff that matters.

A comment on A search engine that favors text-heavy sites and punishes modern web design. Parent comment

Re:A search engine that favors text-heavy sites and punishes... by dannyw on Thursday September 16, 2021 @09:18AM (#28551373)
Not OP, but if I was to do this, I'd start by downloading Wikipedia and all its external links and references, and crawling from there. You should eventually reach most of the publicly visible internet.
[ Parent | on HN ]
Re:A search engine that favors text-heavy sites and punishes... by c0wb0yc0d3r on Thursday September 16, 2021 @09:50AM (#28551760)
I feel a little embarrassed that I didn't think of something like that.

When I did some crawler experimenting in my younger years, I thought I was pretty clever using sites that would let you perform a random Google searches. I would just crawl all the pages from the results returned.

Your method would undoubtedly be more interesting I think. It would certainly lead to interesting performance problems quicker, I bet.

[ Parent | on HN ]