How to write a crawler

Re:How to write a crawler by staunch on Sunday September 29, 2013 @06:46PM (#6466853)

The only complicated part that I've run into when writing crawlers is accidental tarpits. It's very easy to run into a situation in which you're repeatedly requesting the same content via many different URLs.

For example, when a tracking parameter is added to any URL within the site:

http://example.com/?cid=104484&pid=12002348&ref=1294902

http://example.com/?cid=104484&pid=12002348&ref=1294904

http://example.com/?cid=104484&pid=12002348&ref=1294905

http://example.com/?cid=104484&pid=12002348&ref=1294906

You can quickly get to billions of permutations for a single site. The canonical tag solves the problem when it's there, but I still haven't seen a simple solution to the problem when it's not.

[ Parent | on HN ]

Re:How to write a crawler by EmanueleMinotto on Sunday September 29, 2013 @07:03PM (#6466912)

If it's not included in robots.txt rules and doesn't have a canonical link that's not a problem, because the bot can't know if those pages are different or not so those pages you linked are different. This is the reason why crawlers can't try to fill forms.

If you are really sure that these pages are the same, try checking the body content (if two or more pages have the same MD5 of the content, those pages are the same) or look for a form that generate those URLs.

[ Parent | on HN ]

Re:How to write a crawler by novaleaf on Monday September 30, 2013 @09:05AM (#6469341)

great tip on the md5, thanks

[ Parent | on HN ]

Re:How to write a crawler by TheLoneWolfling on Sunday September 29, 2013 @08:24PM (#6467147)

Store a hash of each file you already have. If too many items from a single website collide, throw up a warning/error/flag for help/use some fuzzy method of identifying clashing URLs.

If that's too slow/space-intensive, try a bloom filter.

[ Parent | on HN ]