Re:Scrape like the big boys
by Andoryuuta on Friday November 05, 2021 @12:20PM
(#29121406)
I'm really surprised that the JS challenges helped so much, given that there are open source libraries for bypassing them (e.g. cloudscraper[0]).
If someone wanted to get past it they probably could. We've had a few sources of traffic that we've had to straight up block (as opposed to challenge) because of this exact issue. So far it's been a "good enough" solution that blocks enough of the bot traffic to be effective.
Re:Scrape like the big boys
by EdwardDiego on Friday November 05, 2021 @10:51AM
(#29120101)
What were they scraping, if I can ask? Was it targeted or just wget -r style?
It was a hybrid of low-effort vulnerability scanning and targeted inventory scraping. Many dealerships in the automotive space will pay gray-hat third parties to scrape and compile data on their competitors.
The irony for us as a provider is that it's one of our customers (party A) paying a third party to scrape data from another one of our customers (party B) which in turn affects the performance of party A's site. We've started blocking these third parties and directing them to paid APIs that we offer.
And how do you get your 'inventory data'? Aren't you scraping (or using scraped data) yourself? Oh the irony :)
No, we're a contracted provider for these customers. They ingest their data into our network through APIs or CSVs.
Makes little sense - customers upload data to you and they don't want any data back? Really?
It's not them who want it back, it's their competitors who want it.
I get it why someone else scrapes it. But why customers upload data in the first place? Aren't they interested in getting some OTHER data from you and that OTHER data may as well be scraped?
We chose to switch to the JS challenge screen as it requires no human interaction. We now block 75% (estimated to the best of our knowledge) of bot traffic but some customers are livid over the challenge screen.