News for hackers. Stuff that matters.
Re:How to crawl a quarter billion webpages in 40 hours (2012) by onion2k on Thursday June 15, 2023 @05:46AM (#36338261)
To save you a click: Use 20 machines.
Re:How to crawl a quarter billion webpages in 40 hours (2012) by sethammons on Thursday June 15, 2023 @11:39AM (#36342067)
250M / 40hrs / 60min / 60s ~= 1,737 rps. That over 20 machines is ~87 rps per machine.

Depending on a few factors, I rough out my backend Go stuff to handle between 1-5k rps per machine before we have real numbers.

Re:How to crawl a quarter billion webpages in 40 hours (2012) by onion2k on Thursday June 15, 2023 @03:38PM (#36345416)
You didn't write your Go stuff 11 years ago though...
Re:How to crawl a quarter billion webpages in 40 hours (2012) by sethammons on Thursday June 15, 2023 @06:07PM (#36347322)
We started with Go version 1.2 which was released over ten years ago. Pretty darn close to 11 years.

https://go.dev/doc/go1.2

Re:How to crawl a quarter billion webpages in 40 hours (2012) by haggy102 on Saturday June 17, 2023 @11:40PM (#36377094)
Wow crazy how time flies. I did not realize Go has been in production for over 10 years now.