News for hackers. Stuff that matters.
Re:A search engine that favors text-heavy sites and punishes... by c0wb0yc0d3r on Thursday September 16, 2021 @09:15AM (#28551332)
How did you go about seeding your web crawler with URLs to crawl?
Re:A search engine that favors text-heavy sites and punishes... by marginalia_nu on Thursday September 16, 2021 @09:31AM (#28551541)
I just started with my website and did a crawl. Subsequently I've been seeding it with the best results form my previous crawls.

It's a directed search so it doesn't seem to need a particularly solid seed to get decent results.

Re:A search engine that favors text-heavy sites and punishes... by c0wb0yc0d3r on Thursday September 16, 2021 @09:39AM (#28551629)
So how long did it take to get to 17 million URLs?
Re:A search engine that favors text-heavy sites and punishes... by dannyw on Thursday September 16, 2021 @09:18AM (#28551373)
Not OP, but if I was to do this, I'd start by downloading Wikipedia and all its external links and references, and crawling from there. You should eventually reach most of the publicly visible internet.
Re:A search engine that favors text-heavy sites and punishes... by c0wb0yc0d3r on Thursday September 16, 2021 @09:50AM (#28551760)
I feel a little embarrassed that I didn't think of something like that.

When I did some crawler experimenting in my younger years, I thought I was pretty clever using sites that would let you perform a random Google searches. I would just crawl all the pages from the results returned.

Your method would undoubtedly be more interesting I think. It would certainly lead to interesting performance problems quicker, I bet.