Re:The Evolution of Marginalia's Crawling
by marginalia_nu on Tuesday August 23, 2022 @04:50PM
(#32571308)
I'll crawl those sites too. Most of them will be weeded out in the processing stage, where I exclude websites that have too much heavy duty javascript and tracking and so on. I only index about 20% of the documents I fetch.
Some still slip by, but the ranking algorithm takes care of the rest. I'm using personalized pagerank[1] biased toward a set of real human websites, which turns out to rather aggressively promote human websites.
Some still slip by, but the ranking algorithm takes care of the rest. I'm using personalized pagerank[1] biased toward a set of real human websites, which turns out to rather aggressively promote human websites.
[1] http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf (see ch. 6)