Re:Bypassing website anti-scraping protections
by bmpafa on Friday June 01, 2018 @11:26AM
(#17206178)
As I understand it, this was the rationale behind the courts' decision to prohibit LinkedIn from banning people from scraping public profiles.
Basically it was anti competitive to grant certain privileges to major players around 'public data,' but to block smaller players.
No telling if/when ramifications from that decision (last year) hit existing anti scraping measures, though.
They cannot. Googlebot & some other search engine bots (like Bing's & Yandex's) get special treatment in various websites. This includes things like ban on non-whitelisted scrapers & bypassing paywalls. If you are not already established player in the field, you would not get able to scrape the same websites as the established players can.