Trolling the search engines

Re:Trolling the search engines by xavian on Sunday March 16, 2014 @05:34PM (#7411419)

I maintain web crawlers for a large internet portal and this stuff is a pain enough to deal with when people unintentionally make their websites recursive, much less INTENTIONALLY. facepalm At least google crawler had issues as well. Thanks a lot, troll. ;P

[ Parent | on HN ]

Re:Trolling the search engines by arbuge on Monday March 17, 2014 @09:13AM (#7414801)

You need to update those crawlers to enforce some hard limits on crawl depth, methinks, especially for low pagerank domains (if there's some easy way to estimate pagerank, i.e.).

[ Parent | on HN ]