Re:How to crawl a quarter billion webpages in 40 hours (2012)
by onion2k on Thursday June 15, 2023 @03:38PM
(#36345416)
You didn't write your Go stuff 11 years ago though...
Re:How to crawl a quarter billion webpages in 40 hours (2012)
by sethammons on Thursday June 15, 2023 @06:07PM
(#36347322)
We started with Go version
1.2 which was released over ten years ago. Pretty darn close to 11 years.
Depending on a few factors, I rough out my backend Go stuff to handle between 1-5k rps per machine before we have real numbers.