Re:How to crawl a quarter billion webpages in 40 hours (2012)
by ricardo81 on Thursday June 15, 2023 @10:53AM
(#36341360)
It makes very little difference what IP you scrape from, unless you're from a very dodgy subnet.
The major content providers tend to go on a whitelist only based approach, you're either a human-like visitor or facing their anti-scraping methodologies.
The major content providers tend to go on a whitelist only based approach, you're either a human-like visitor or facing their anti-scraping methodologies.