Re:How to crawl a quarter billion webpages in 40 hours (2012)
by onion2k on Thursday June 15, 2023 @05:46AM
(#36338261)
To save you a click: Use 20 machines.
[ Parent
| on HN ]
Re:How to crawl a quarter billion webpages in 40 hours (2012)
by sethammons on Thursday June 15, 2023 @11:39AM
(#36342067)
250M / 40hrs / 60min / 60s ~= 1,737 rps. That over 20 machines is ~87 rps per machine.
Depending on a few factors, I rough out my backend Go stuff to handle between 1-5k rps per machine before we have real numbers.
Re:How to crawl a quarter billion webpages in 40 hours (2012)
by onion2k on Thursday June 15, 2023 @03:38PM
(#36345416)
You didn't write your Go stuff 11 years ago though...
Re:How to crawl a quarter billion webpages in 40 hours (2012)
by sethammons on Thursday June 15, 2023 @06:07PM
(#36347322)
We started with Go version
1.2 which was released over ten years ago. Pretty darn close to 11 years.