News for hackers. Stuff that matters.

A comment on Avoiding Webscraping Throttling Using Python and Tor as a Proxy. Parent comment

Re:Avoiding Webscraping Throttling Using Python and Tor as a... by evilDagmar on Monday July 15, 2019 @12:21AM (#20438092)
It typically becomes rather "challenging" to justify to management why one shouldn't scrape "aggressively" until the competitor's site degrades or falls over. That's why it's generally kind of sleazy.
[ Parent | on HN ]
Re:Avoiding Webscraping Throttling Using Python and Tor as a... by bydl0coder on Monday July 15, 2019 @02:56AM (#20438691)
Large eCommerce sites are constantly scraped by competitors, market research companies and even vendors (to ensure sellers honor MSRP). Scraping activity is usually throttled at reasonable levels.
[ Parent | on HN ]
Re:Avoiding Webscraping Throttling Using Python and Tor as a... by FDSGSG on Monday July 15, 2019 @11:23AM (#20441659)
Could you tell us more about your "typical" experiences?
[ Parent | on HN ]