News for hackers. Stuff that matters.
Re:It is not possible to detect and block Chrome headless by merb on Thursday January 18, 2018 @01:13PM (#16179657)
if you want to block scrapers, just add rate limiting...
Re:It is not possible to detect and block Chrome headless by jjeaff on Thursday January 18, 2018 @01:23PM (#16179772)
That's old hat and ineffective. Scrapers usually proxy through large lists of rotating ip addresses. There are lots of services for it.
Re:It is not possible to detect and block Chrome headless by saas_co_de on Thursday January 18, 2018 @02:46PM (#16180644)
depends. if you also do tracking for valid navigation paths then rate limiting may be effective.

For instance: if you have a search result with a 1000 pages that someone is trying to scrape if you don't allow people to jump into the middle of the result set then just rotating IPs doesn't work.

Re:It is not possible to detect and block Chrome headless by lr4444lr on Thursday January 18, 2018 @01:19PM (#16179734)
I'll eat you through a proxy network then, unless you want to slow down your legitimate users too.
Re:It is not possible to detect and block Chrome headless by thaumaturgy on Thursday January 18, 2018 @01:27PM (#16179815)
I now work for a company that is gathering metadata on the IP address space (in an effort to reduce the amount of abuse that sites and service providers have to deal with).

It won't be very long before it'll be possible to identify most of the common proxying networks and block those.

Scrapers can respond by setting up something like an ssh tunnel from a residential high speed connection to a remote server (so that scraping run on the remote server appears to come from the residential connection) and using that as a proxy, but it adds another level of effort.

(I am generally on the scrapers' side on this and think it's ultimately futile to try to block all scrapers. Website administrators need to accept that anything they put on the internet is public and get over it.)

Re:It is not possible to detect and block Chrome headless by lr4444lr on Thursday January 18, 2018 @02:19PM (#16180382)
Just out of curiosity, how are you going to handle Luminati? They're bot-networking home users all over the world in exchange for free VPN.
Re:It is not possible to detect and block Chrome headless by thaumaturgy on Thursday January 18, 2018 @06:19PM (#16182554)
That's pretty neat.

For our purposes, those residential IPs that are used maliciously through this service will just hurt the reputation of the ISP they belong to. I suspect we'll see something shake out in the data where this activity is limited to some kind of specific demographics (ISP, netrange, geographic location) and shouldn't interfere too badly with the system as a whole.

Re:It is not possible to detect and block Chrome headless by lr4444lr on Thursday January 18, 2018 @07:29PM (#16183008)
That's what I thought. In case you were wondering, as a Luminati user (on the proxy end, not the VPN end thank god), my experience has been higher latency and request fail rate than on traditional proxy networks. It's definitely not my first weapon in the arsenal, but since the target for multiple reasons already has an effective high latency and my users are sufficiently motivated enough to wait for what I'm delivering, it's a pretty effective last resort.
Re:It is not possible to detect and block Chrome headless by saas_co_de on Thursday January 18, 2018 @02:43PM (#16180616)
wow. hadn't seen this before.
Re:It is not possible to detect and block Chrome headless by Jach on Thursday January 18, 2018 @01:44PM (#16180004)
Maybe companies can instead focus on improving network infrastructure and software architecture to be able to eat any amount of scraper traffic since at some point it must become indistinguishable from a ddos that also needs to be handled...
Re:It is not possible to detect and block Chrome headless by tehlike on Thursday January 18, 2018 @02:03PM (#16180194)
Its not about ddos. Think amazon. They probably dont want to reveal all of heir inventory, their price history and so on. Even their api had rate limiting per second, and i doubts it is because of the load
Re:It is not possible to detect and block Chrome headless by almost_usual on Thursday January 18, 2018 @01:22PM (#16179761)
There is a multi-million dollar industry around blocking scrapers, it's not "just add rate limiting".
Re:It is not possible to detect and block Chrome headless by kowdermeister on Thursday January 18, 2018 @01:20PM (#16179749)

    while ScraperBlocked:
        scraper.sleep(5)