News for hackers. Stuff that matters.
Re:Bypassing website anti-scraping protections by learc83 on Friday June 01, 2018 @10:42AM (#17205699)
A company I consulted for was using a paid API to handle search.

Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature.

They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior.

Everytime I implemented a new countermeasure they'd try to find a way around it. It was maddening because we made everything available for them through the XML feed. They just wouldn't use it.

Re:Bypassing website anti-scraping protections by dredmorbius on Friday June 01, 2018 @12:21PM (#17206914)
An explicit message to use that feed as part of the countermeasures might be useful. Did/do you do this?
Re:Bypassing website anti-scraping protections by jardah on Friday June 01, 2018 @11:03AM (#17205905)
That is kinda sad to hear. The approach should always be to go through the path of least resistance and smallest effect on the website. So for example, if a company has API that can be used instead of scraping their website, then it's always preferred to use the API. Same would go for the XML you mentioned.

It's bad that not everyone works like this; there are quite a lot of people who would rather brute-force a solution than think about it.

Re:Bypassing website anti-scraping protections by greglindahl on Friday June 01, 2018 @01:24PM (#17207706)
The path of least resistance for the bots appears to be that they have a tool that scrapes search results, and nothing to talk to an API.
Re:Bypassing website anti-scraping protections by sheeshkebab on Saturday June 02, 2018 @08:41AM (#17214308)
Interestingly XML is not easy to parse with a lot of these scraping tools that rely on JavaScript... sure, the tools can easily parse HTML and convert to json or csv, but taking xml in random format and doing the same is rather difficult.

It may have been better to just publish the site in HTML format with an easy to find link on front page to access it.

Re:Bypassing website anti-scraping protections by MightySCollins on Friday June 01, 2018 @06:27PM (#17211018)
We have kind of the same thing. All data is in an API which is less than a cheap VPS and display messages to them about using the API if they get blocked but they just come back with a new IP every time.
Re:Bypassing website anti-scraping protections by unreal37 on Friday June 01, 2018 @02:00PM (#17208154)
You had a paid API, and people wanted the information for free....

Not unexpected I guess.

Re:Bypassing website anti-scraping protections by bo1024 on Friday June 01, 2018 @03:52PM (#17209432)
After searching "algolia" mentioned below, I figured out the misunderstanding. The company was paying somebody else per search made on their web site. So every time a scraper called the website's search function, it cost the website money.
Re:Bypassing website anti-scraping protections by mlevental on Friday June 01, 2018 @02:52PM (#17208747)
you've misunderstood: the site used something like algolia (which the site paid for) to index. the scrapers were hitting that service (which was costing the site) rather than parsing the xml (which already had everything).
Re:Bypassing website anti-scraping protections by learc83 on Friday June 01, 2018 @03:01PM (#17208826)
The paid API was just to handle the search of the company database (couldn't change that for political reasons).

They weren't getting any information that they couldn't get through the XML.

Re:Bypassing website anti-scraping protections by ryanwaggoner on Friday June 01, 2018 @01:51PM (#17208034)
Maybe it was sabotage.