Re:Avoiding Webscraping Throttling Using Python and Tor as a...
by chii on Sunday July 14, 2019 @06:36AM
(#20432736)
> While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds
Why is it wrong to scrape your competitor's site for pricing info? If it's publically accessible, it's free game. As long as such scraping doesn't cause any undue DOS'ing or prevent anyone else from accessing such info.
Re:Avoiding Webscraping Throttling Using Python and Tor as a...
by evilDagmar on Monday July 15, 2019 @12:21AM
(#20438092)
It typically becomes rather "challenging" to justify to management why one shouldn't scrape "aggressively" until the competitor's site degrades or falls over. That's why it's generally kind of sleazy.
Re:Avoiding Webscraping Throttling Using Python and Tor as a...
by bydl0coder on Monday July 15, 2019 @02:56AM
(#20438691)
Large eCommerce sites are constantly scraped by competitors, market research companies and even vendors (to ensure sellers honor MSRP). Scraping activity is usually throttled at reasonable levels.
Precisely. While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds, I have no qualms about scraping in certain situations. A couple that spring to mind:
- Where the site's UI is so dumb that it doesn't let me do what any reasonable person might naturally want to do (this would be the case in the original article)
- Where it's a govt. site and the data is only trickled out through some 90's era search function. Taxpayers paid for the collection of that data, taxpayers pay for its storage etc - give the taxpayers an API to access it!