Re:The Evolution of Marginalia's Crawling
by marginalia_nu on Tuesday August 23, 2022 @11:49AM
(#32566687)
In practice, maybe 40-50rps (peaking at 100) for the first design, and 300rps for the second.
Although I'm serving search engine traffic from the same machine, so I'm trying to leave ample bandwidth for that. If I go too fast the NAT starts dropping packets and refusing connections, and that's not great for crawling or serving.
Although I'm serving search engine traffic from the same machine, so I'm trying to leave ample bandwidth for that. If I go too fast the NAT starts dropping packets and refusing connections, and that's not great for crawling or serving.