News for hackers. Stuff that matters.
Re:Indexing a Billion Pages by jetrink on Saturday December 23, 2023 @02:58PM (#38747567)
> We’ve indexed over 100 million pages

> [W]e’re crawling up to a million pages a day, as you can see on our stats page.

> Given that Mwmbl is still relatively unknown, it seems plausible that we can reach our target of crawling three billion pages a day, to refresh the entire index in one month.

I think this is supposed to read "it seems plausible that we can reach our target of crawling three million pages a day."

Re:Indexing a Billion Pages by hinkley on Saturday December 23, 2023 @11:37PM (#38751173)
> to refresh the entire index [of 1 billion] in one month.

Seems you are correct.

Re:Indexing a Billion Pages by halfdan on Sunday December 24, 2023 @07:13AM (#38752965)
Except then it needs to be 30M pages/day. 3M * 30 days just nets you 90M