News for hackers. Stuff that matters.
Re:How to crawl a quarter billion webpages in 40 hours by hntester123 on Saturday August 11, 2012 @06:26PM (#4371168)
>I'm rather impressed that search engines do it so well. I imagine the right approach involves examining the contents of the pages and doing checksums

Interesting point. Q: Is it possible for two different pages to give the same checksum? (Asking for my own info; I do know what checksums are, in overview, from way back, but haven't checked :) them out much.

Re:How to crawl a quarter billion webpages in 40 hours by josephlord on Saturday August 11, 2012 @07:01PM (#4371243)
Given two different pages getting the same checksum should be VERY unlikely. Getting a collision among billions of pages is fairly or very likely depending on the type of checksum. So it is likely to be important to only compare within domains or divide the space by some method.

Also it doesn't help you at all because pages returned that you want to treat as the same aren't actually the same because they include the request time or other unique element.

Edit: Added sentence about comparing with domains.

Re:How to crawl a quarter billion webpages in 40 hours by mrkrwtsn on Saturday August 11, 2012 @11:53PM (#4371763)
You could use an edit distance algorithm instead of using checksums. Although that would be really time intensive.