Re:How to crawl a quarter billion webpages in 40 hours
by cypherpunks01 on Friday August 10, 2012 @04:51PM
(#4368186)
You're right if OP was using a small bloom filter, and wrong if it was a big one. Hence, the phrase, "depending how you tune it."
Re:How to crawl a quarter billion webpages in 40 hours
by JoshTriplett on Friday August 10, 2012 @04:47PM
(#4368169)
You can construct a bloom filter with an arbitrarily low error rate.
My (not-very fresh) memory of what bloom filter actually is tells me that this "non-zero rate" you're talking about must be HUUUUGE. In order of millions of pages. Am I right?