Re:How to crawl a quarter billion webpages in 40 hours
by pooriaazimi on Friday August 10, 2012 @04:43PM
(#4368156)
> but you'll have a non-zero rate of pages you think you've downloaded but you actually haven't, depending how you tune it.
My (not-very fresh) memory of what bloom filter actually is tells me that this "non-zero rate" you're talking about must be HUUUUGE. In order of millions of pages. Am I right?
Re:How to crawl a quarter billion webpages in 40 hours
by cypherpunks01 on Friday August 10, 2012 @04:51PM
(#4368186)
You're right if OP was using a small bloom filter, and wrong if it was a big one. Hence, the phrase, "depending how you tune it."
Re:How to crawl a quarter billion webpages in 40 hours
by JoshTriplett on Friday August 10, 2012 @04:47PM
(#4368169)
You can construct a bloom filter with an arbitrarily low error rate.
other way round? bloom filter provides a low probability of erroneously believing a URL had already been added when it had not, zero probability of believing a URL had not already been added to the filter when in fact it had.
using a bloom filter in this way guarantees you won't ever hit a page twice, but you'll have a non-zero rate of pages you think you've downloaded but you actually haven't, depending how you tune it.