There are span sites with more incoming links than Wikipedia....
Designing crawl policy is a lot like designing security policy. The opposition is very well-funded, smart, experienced, and very motivated.
If you're just smart, they've got you beat.
Google apparently solve this with a domain trust factor discounting link juice from low value domains for SERPs purposes but I'm not sure what they feed forward to control their bots.
This means that for some time it has appeared that large numbers of low quality inbound links has been a negative for Google SERP placement ... so I'm wondering if such [spam] domain owners are being as smart as you think??
It's been several years since I worked on Yahoo's crawler.
It's interesting that at the beginning of this, you doubted that spam domains could have large number of inbound links (and suggested that inbound link counts was a powerful indicator) but by the end, you seemed aware of such domains....
I'm not. I've never done any research on "spam domains". Do you have any examples or not?
Then who wrote
> This means that for some time it has appeared that large numbers of low quality inbound links has been a negative for Google SERP placement
?
But none of that presupposes any knowledge of "spam sites", nor of the particular instances you note of spam sites with lots of inbound links (ie the claimed far more than wikipedia [per IP address]).
But it seems you don't have any examples so I'm not sure why you're persisting. Even if I did know about specific spam sites with lots of incoming links I don't see the relevance of my knowing that to you answering the question of if you can give any examples of sites with your claimed characteristics. The points are orthogonal. Your pre-knowledge of such sites is not in any way bound by my knowledge of such sites.
I daren't ask if you can answer the question again. But just suppose you could then a response would still be of interest.
It's probably best to add an IP based limit factor too to handle link farms.
Something like "max-pages crawled is directly proportional to the (incoming links / linking unique domains)"; or just have it naively proportional to the linking unique domains to reduce processing cost.