News for hackers. Stuff that matters.
Re:It is not possible to detect and block Chrome headless by kbenson on Thursday January 18, 2018 @01:38PM (#16179942)
All the passive techniques are much harder to reasons about, but much easier to match. You just look at the complete request/response headers, make sure you match them, and have some good sources to request from.

Much harder is stuff like Distil's script injection, where they transparently inject script tags that do fingerprinting, and they obfuscate the code that does so annoyingly (it's not really hard to reverse, just time consuming and and annoying). They pair this with being a bit more user friendly by redirecting you to a CAPTCHA page if your fingerprinting hits some threshold which if you answer redirects you to the page you wanted, so users experience and inconvenience if there's a false positive, but still get access to what they wanted.

I was able to get around most of the passive fairly easily with Perl and LWP, and even the active stuff and CAPTCHA redirects (cookie_jar all serialized to DB so I could store request and represent it to a user to answer), but once they started tweaking their fingerprint script ever couple months/weeks that's when the equation shifted. Distil, as a solutions provider, gets to amortize their changes across all their customers, while I would have to spend the time de-obfuscating it. They could just assign a person to change it once a week and they would effectively halve my time to get any real work done, so without a collective effort of some sorts to combat them, I saw the writing on the wall. :/

The sad thing is that when we moved to API access, their APIs are hampered to the degree that it actually takes two orders of magnitude more requests each minute for a fraction of the accuracy (I was able to query changes over the last couple minutes previously, and now I have to query the entire item set of a subset of all containers, when there are tens of thousands of containers). :/ Lose lose, since our use case isn't even the main reason the site wanted to block scrapers.

Re:It is not possible to detect and block Chrome headless by ThrustVectoring on Thursday January 18, 2018 @02:17PM (#16180357)
Did you do a cost estimate for the "Wizard of Oz" solution of having real people with real browsers (and a script to pull data from the site?) Might have been worthwhile.
Re:It is not possible to detect and block Chrome headless by kbenson on Thursday January 18, 2018 @02:57PM (#16180735)
It was actually two changes, one which required a much more intensive request regime because of a public caching (SOLR) system change, and then the much more aggressive scraping detection. The first caused us to change from requesting data 78 times an hour (items changed within 2 minutes requested every minute, 6 minutes of changes every 5 minutes, 11 minutes of changes every 10 minutes for overlapping coverage) to many thousands of checks an hour for much less accurate information. In the end, we ended up doing very targeted checks and much less accuracy for different classes of items. Having people actually do the checking just wouldn't be feasible for our size and resources (very small company, <10 employees), even through mechanical turk (I suspect).