News for hackers. Stuff that matters.
Re:The Evolution of Marginalia's Crawling by marginalia_nu on Tuesday August 23, 2022 @04:50PM (#32571308)
I'll crawl those sites too. Most of them will be weeded out in the processing stage, where I exclude websites that have too much heavy duty javascript and tracking and so on. I only index about 20% of the documents I fetch.

Some still slip by, but the ranking algorithm takes care of the rest. I'm using personalized pagerank[1] biased toward a set of real human websites, which turns out to rather aggressively promote human websites.

[1] http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf (see ch. 6)

There are no comments on this comment yet.