Re:The Evolution of Marginalia's Crawling
by stinkytaco on Tuesday August 23, 2022 @04:45PM
(#32571229)
How do you keep it "indie"? Even fairly small sites probably link to Forbes or the Atlantic once in a while. Do you have a specific "block" list to keep large, commercial sites out of the results?
[ Parent
| on HN ]
Re:The Evolution of Marginalia's Crawling
by marginalia_nu on Tuesday August 23, 2022 @04:50PM
(#32571308)
I'll crawl those sites too. Most of them will be weeded out in the processing stage, where I exclude websites that have too much heavy duty javascript and tracking and so on. I only index about 20% of the documents I fetch.
Some still slip by, but the ranking algorithm takes care of the rest. I'm using personalized pagerank[1] biased toward a set of real human websites, which turns out to rather aggressively promote human websites.
[1] http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf (see ch. 6)