News for hackers. Stuff that matters.
Re:Detecting Chrome headless by stevefeinstein on Saturday August 05, 2017 @11:33AM (#14936194)
So again someone wants to punish all the legitimate people using a web site to get some marginal benefit from detecting the remaining <1%. The inevitable false positives don't affect the "malicious" users. Only the legitimate ones. And how much will this bloat the page load by? Adding more code to an already overly large page isn't helping anyone.

Just let the web be the web, and stop trying to control it.

Re:Detecting Chrome headless by tyingq on Saturday August 05, 2017 @12:37PM (#14936532)
Mentioned this in another comment, but for some websites, the scraping problem has real costs. Airline, hotel, stock prices,etc. For some spaces, scaling and paying bandwidth for unconstrained scraping is costly. And not restricting it hurts the legitimate users because the performance sucks.

There are also the scrapers blindly looking for vulnerabilities or other unsavory tactics.

Re:Detecting Chrome headless by BinaryIdiot on Saturday August 05, 2017 @10:53PM (#14939428)
While I agree with you to a degree most airlines and hotels have APIs that can be consumed to get pricing information there are just restrictions in what you can do with that information.

Not sure about stock prices (I think it's pretty common to pay for real time data there?).

But I can certainly see sites that have a lot of data for their users facing major bandwidth costs if a lot of people were scraping their data. This type of detection isn't really an answer for that, though, as it's easy to mitigate for a scrapper.

Re:Detecting Chrome headless by always_good on Saturday August 05, 2017 @02:17PM (#14937063)
Also, what I've learned is how little regard for your site your scrapers often have, scraping as aggressively as possible.

You're just not always in a place to scale to the abuse or build something more complex than some simple heuristic filters.

Re:Detecting Chrome headless by josteink on Sunday August 06, 2017 @09:49AM (#14941298)
> what I've learned is how little regard for your site your scrapers often have, scraping as aggressively as possible.

Often? Based on what data?

I find it much more likely you only often notice aggressive scrapers. That however tells you nothing about the behavior of the average web scraper or web scrapers in general.

Re:Detecting Chrome headless by tyingq on Sunday August 06, 2017 @12:04PM (#14941824)
The system encourages it. Ingress data is cheap, and so many scrapers just default to high frequency.
Re:Detecting Chrome headless by mdominguez on Saturday August 05, 2017 @02:21PM (#14937082)
You're not taking into consideration the economical consequences some bots have on companies. Some bots are designed to make payments with fake or stolen credit cards. Some bots impact on people that need to manually check for submissions or takedown notices. Obviously I agree that there are legitimate use for bots and scrapers, but that admittedly low percentage of fraudulent use cases do cause a lot of harm.
Re:Detecting Chrome headless by walterstucco on Saturday August 05, 2017 @03:06PM (#14937320)
Some bot can and will use a real browser, in a real window, opening many sessions, randomising usage patterns to look more human, and continue doing what they already do.
Re:Detecting Chrome headless by chii on Saturday August 05, 2017 @11:46PM (#14939557)
if a human is allowed to do something on a site, it goes to reason that a bot should be allowed too (granted using the same access frequency as a human).

Blocking scraping is like DRM. Don't do it. Use a legal mechanism to deal with copyright infringement, and use acceptable usage policy to deal with heavy users that are using more than their "fair" share of bandwidth.

Re:Detecting Chrome headless by kuschku on Saturday August 05, 2017 @06:25PM (#14938269)
Some people just hire actual humans to do this, for cheap.