technokrat233

Having worked with/on two large-scale web crawling systems at FAST Search & Transfer; The C-Crawler that powered alltheweb.com (owned now by Yahoo) and a python-based crawler that was a workhorse for FAST ESP/Scirus.com/etc.. (now owned by Microsoft); this run-down is pretty good. Some things I'd add:

- Link traps. If you limit to 100 pages per site not as big an issue but if you want to go deeper you need a way to detect when a site is generating garbage.

- Near duplicate detection. There's lots of sites like you mention that republish content of others, but some just present it in different ways with different headers, timestamps, etc..

- Content/meta-data detection/extraction, once crawled you want to do something with the content and detecting the actual content of pages is non-trivial if you don't want headers/ads/etc..

- How do you handle non-HTML content (PDF, Docs, etc?)

- How do you handle large content (sample, truncate, ignore)?