News for hackers. Stuff that matters.
Re:Bypassing website anti-scraping protections by unreal37 on Friday June 01, 2018 @02:00PM (#17208154)
You had a paid API, and people wanted the information for free....

Not unexpected I guess.

Re:Bypassing website anti-scraping protections by bo1024 on Friday June 01, 2018 @03:52PM (#17209432)
After searching "algolia" mentioned below, I figured out the misunderstanding. The company was paying somebody else per search made on their web site. So every time a scraper called the website's search function, it cost the website money.
Re:Bypassing website anti-scraping protections by mlevental on Friday June 01, 2018 @02:52PM (#17208747)
you've misunderstood: the site used something like algolia (which the site paid for) to index. the scrapers were hitting that service (which was costing the site) rather than parsing the xml (which already had everything).
Re:Bypassing website anti-scraping protections by learc83 on Friday June 01, 2018 @03:01PM (#17208826)
The paid API was just to handle the search of the company database (couldn't change that for political reasons).

They weren't getting any information that they couldn't get through the XML.