News for hackers. Stuff that matters.
Re:Detecting Chrome headless by dsacco on Saturday August 05, 2017 @11:35AM (#14936206)
Interesting. I've done a lot of scraping and I never did this. Do you have a treatise that explains how it's done? I've actually been experimenting with headless Chrome recently because I like it more than PhantomJS.

Practically speaking I don't think IP detection is useful at all these days, and the only Captcha that can't be bypassed is Google's most recent version. The much more successful anti-scraping tactic is to use a sophisticated reverse proxy that analyzes all requests to identify patterns and unusual behavior, indiscriminate from the IP source (because any large scale scraping will come from many IPs anyway).

Re:Detecting Chrome headless by pebers on Saturday August 05, 2017 @12:03PM (#14936336)
It's not terribly hard using Selenium; you can use one of their Docker images (e.g. selenium/standalone-chrome) and connect to that using Webdriver.

Having said that, it has flaws compared to headless Chrome (more moving parts, terrible security, extra memory usage / dependencies for the JVM to run Selenium) so unless that camouflage is proving crucial, I'd avoid it where possible. I've recently migrated a project to headless Chrome and definitely prefer it.

Re:Detecting Chrome headless by taeric on Saturday August 05, 2017 @02:37PM (#14937182)
I'm curious on what makes the security worse. The rest, I can see. Though, I have to confess I have been more critical of the whole headless browser binary push than I'd care to acknowledge. I just feel like this is solving a problem that didn't actually exist anymore. Selenium has worked solidly for quite a while. And, you almost certainly should test in multiple browsers/scenarios.
Re:Detecting Chrome headless by pebers on Saturday August 05, 2017 @03:33PM (#14937454)
The main security issues I know of so far are:

  - Neither Selenium nor Webdriver (at least the Python client, but I assume others too) support HTTPS at all.
  - By default it opens browsers configured to accept any SSL certificate. I can see why that's useful for local testing, but it's a terrible choice to default to.
  - It logs way too much. Logs every keypress sent to it, including passwords.
At least the latter two are fixable, but not trivially so, and better defaults with easy client-side options to opt into insecure mode would have been much better. The former is not fixable without patching both the client and server and in 2017 it's pretty poor to not even have an option for.
Re:Detecting Chrome headless by taeric on Saturday August 05, 2017 @05:31PM (#14938028)
Odd, I'm pretty sure both of the first two are wrong. We used to have to make sure we had legit certs for selenium to work with https. That or configure the Firefox profile to have already accepted the self signed one.

And, really, the first two points are clearly contradictory. I'm guessing I just misunderstand what you mean?

Do you mean the communication between the driver? I'm curious why ssl would be important there? Should just be done locally and a standard ssh tunnel can help with any remote encryption you might want.

Re:Detecting Chrome headless by pebers on Sunday August 06, 2017 @06:22AM (#14940726)
The first two are not contradictory, the first refers to communication between the client and server and the second to the browser's communication with a remote server. Maybe that wasn't clear from how I wrote it.

On further investigation, I think I was wrong on the second though; it might be that it's built into Chromedriver, not Selenium, which would explain why you didn't have the same issue with Firefox.

SSL between the client & Selenium server would be of benefit to keep any bad actor on the network from man-in-the-middle attacks on it. I'm not familiar with how I'd set up an ssh tunnel for that, I'm happy to believe it can be done, but it'd be a lot easier if it just supported HTTPS to begin with.

Re:Detecting Chrome headless by taeric on Tuesday August 08, 2017 @01:28PM (#14960161)
Apologies, I should have relaxed more of my verbiage. I was writing on my phone, and only about halfway through my post did it "click" what you meant.

HTTPS would be an odd choice, if only because I don't want to do any cert management for my selenium test runner. An encrypted channel makes some sense, but I'm not sure what the best mechanism for the shared secret would be. Ssh kind of gets you there, but is not as straight forward, as you ntoed.

Instead, I'd urge to keep the communication at localhost (or tunneled over SSH, at worst). Preferably with a locked down security model on the network so that you will see any traffic going off.

Re:Detecting Chrome headless by figgis on Saturday August 05, 2017 @09:41PM (#14939205)
We both must be misunderstanding, because those first two points are blatantly false as far as I can tell. Selenium can handle(?) invalid SSL certs but the defaults certainly don't freely accept them.

As for the third point.. That's why we have DMZ's..

Re:Detecting Chrome headless by kuschku on Saturday August 05, 2017 @08:25PM (#14938902)
> and the only Captcha that can't be bypassed is Google's most recent version

That can also be easily bypassed (I’ve done that, without actually planning to do so) if you can make your behaviour seem completely human.

So if you have a bot that can browse in a way that seems statistically human you can actually get around that. You can still scrape the same datasets – you just need to keep instances of bots, with all cookies around, and have them browse in certain ways. Classify what category a site might likely belong to, put them into buckets of queues, and have bots pull from queues they’re likely to browse, or search for certain terms they’re likely to search for.

In my case, this all happened by accidents – I had IRC bots for a few channels, each kept a perpetual session and would visit every site that was linked (to get the title) and would be able to search Google by scraping. One day I was accessing a bot remotely, and told it to access a page that was NoCaptcha protected (because the site wasn’t working on my home system), yet it passed the captchas perfectly fine. Tried a few more times, always worked. So I tried figuring out why it worked.

Re:Detecting Chrome headless by _e on Saturday August 05, 2017 @11:07PM (#14939462)
Were the irc bots able to render javascript or were the bots using a cli lib like python requests?

I thought that a client which doesnt run javascript would be a huge red flag.

Re:Detecting Chrome headless by kuschku on Sunday August 06, 2017 @08:44AM (#14941067)
They used Firefox with off-screen rendering, as otherwise they wouldn’t be able to get the title of many pages – even many blogspot blogs can’t be read without JS anymore.
Re:Detecting Chrome headless by heipei on Saturday August 05, 2017 @11:43AM (#14936235)
If you know how to interface with Chrome headless then you're 95% there already. Just run xvfb, start a Chrome instance on the DISPLAY with the appropriate flags (for exposing the debugging port) and you're done.
Re:Detecting Chrome headless by IgorPartola on Saturday August 05, 2017 @11:45AM (#14936247)
IP detection, especially IPv4, should still work fine because the cost to get a new IP will often outweigh the benefit of the scraping.

Having said that, just let them scrape?

Re:Detecting Chrome headless by jalfresi on Saturday August 05, 2017 @12:32PM (#14936501)
There exists an entire industry that provide rotating IPs across fleets of proxies for very cheap prices. Proxy services like this make it pretty much impossible to prevent scraping.
Re:Detecting Chrome headless by tyingq on Saturday August 05, 2017 @12:30PM (#14936492)
>Having said that, just let them scrape?

Depends on who you are. Hotel and airline websites, for example, are pummeled with scrapers wanting pricing and availability info. Letting them scrape with no limits is costly.

Re:Detecting Chrome headless by IgorPartola on Saturday August 05, 2017 @05:21PM (#14937979)
And why is that a good thing for them? I mean if they provided an API to get their pricing, wouldn't it be in their interest? Does anyone buy airline tickets from the airline website ever? I don't. I use sites like Expedia and Priceline. And if an airline is not listed or has shitty prices, I don't buy it. Wouldn't it be in their interest to be listed on there? Moreover, wouldn't they want the other companies to have similar API's to have dynamic pricing?

I get that at one point not having this info out there was a good thing for them. But now the car is out of the bag. You can't keep pretending that booking sites don't exist.

Re:Detecting Chrome headless by tyingq on Saturday August 05, 2017 @07:48PM (#14938717)
Lots of wannabe sites with heavy scraping, but no sales.

There are several airlines where 50+% of all sales are on their own website. Lowest distribution cost.

Re:Detecting Chrome headless by ValentineC on Sunday August 06, 2017 @12:21PM (#14941911)
> Does anyone buy airline tickets from the airline website ever?

All other things being equal, I will strongly prefer buying directly from the airline, because I've personally experienced the shifting of responsibility if things go wrong during a flight.

I've also been denied boarding (not in my home country) because the airline claimed that the OTA (Netflights) didn't pay for my ticket, leaving me to spend the night in the airport and having to book a one-way flight with a different airline the next morning.

That was a horrible experience I'm determined to not repeat.

Re:Detecting Chrome headless by vacri on Saturday August 05, 2017 @10:19PM (#14939302)
Adding to tyingq's comment, not all scrapers are efficient, either.

Also, if you don't have the rock-bottom cheapest prices, you may not want to cater to aggregators that promote those.

Re:Detecting Chrome headless by mateo411 on Sunday August 06, 2017 @01:57PM (#14942454)
People who have miles or points buy airline tickets directly from the airline.
Re:Detecting Chrome headless by dsacco on Saturday August 05, 2017 @01:17PM (#14936719)
> Hotel and airline websites, for example, are pummeled with scrapers wanting pricing and availability info.

Yes, and they make it very cumbersome to discourage it. I've successfully written scrapers for airlines. It's significantly more difficult than crawling other websites for a few reasons:

1. Session management is wacky - they really like to manage state entirely through cookies, and you typically need to visit a specific set of pages in a specific sequence before you can access the resources you want, like the number of seats available or their prices.

2. Sessions have time limits because anyone who looks at the seats initiates a "soft" reservation on them (this works in a similar way for theatre, concert and movie seating).

3. You don't usually have nice JSON endpoints, so you'll be doing a lot of HTML parsing (which, given the type of HTML you encounter, can be hell).

Re:Detecting Chrome headless by tyingq on Saturday August 05, 2017 @01:24PM (#14936756)
>You don't usually have nice JSON endpoints, so you'll be doing a lot of HTML parsing

That is changing. Most scrapers haven't caught on, but the more modern things airlines are pushing out (their mobile sites and native mobile apps) often have really nice REST/JSON api interfaces. The scrapers are often still scraping the old desktop site which will be the last to get that underpinning.

Re:Detecting Chrome headless by dsacco on Saturday August 05, 2017 @02:47PM (#14937231)
Yes, whenever I'm looking for a source to crawl I prefer mobile applications for precisely this reason. Request signing and certificate pinning are an upfront annoyance, but the maintainability is far higher.
Re:Detecting Chrome headless by heipei on Saturday August 05, 2017 @11:49AM (#14936262)
There are multiple ways to hide your IP for free: Public proxies, VPNs which are not so well-known, and lastly there is a wealth of browser extensions which aim at hiding your IP to some extent. This would be one example where your traffic will exit with actual users on consumer ISP lines: http://hola.org/
Re:Detecting Chrome headless by ficklepickle on Saturday August 05, 2017 @09:01PM (#14939071)
I'd say don't use Hola. They've had some issues, like selling their users bandwidth for botnets[1].

If you do some googling, you will see many reasons not to use that extension.

[1]http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-se...

Re:Detecting Chrome headless by ant6n on Saturday August 05, 2017 @02:01PM (#14936964)
What about tor.
Re:Detecting Chrome headless by heipei on Saturday August 05, 2017 @02:26PM (#14937110)
TOR is an easy one to detect. Case in point: Try to browse the web with TOR and see how many CloudFlare captchas you have to solve in the first 10 minutes ;)

Also TOR has public lists of exit nodes (https://torstatus.blutmagie.de/), and unless you exit via a non-listed one you're trivially identifiable.

Lastly, TOR is rather slow, which prohibits using for large-scale scraping tasks. Plus you get switched around different exits and countries, which might break your scraping logic.

Re:Detecting Chrome headless by crtasm on Sunday August 06, 2017 @01:47PM (#14942401)
I believe you could use the ExitNodes option to only use a specific exit.

There's no such thing as an unlisted exit node, only unlisted entry nodes (bridges). I agree that Tor isn't a good choice for scraping.

Re:Detecting Chrome headless by dsacco on Saturday August 05, 2017 @02:45PM (#14937221)
Tor would introduce pretty intolerable latency into a professional data crawling setup.
Re:Detecting Chrome headless by skinnymuch on Saturday August 05, 2017 @11:50AM (#14936269)
I'm thinking OP meant if you want to never be detected, but like you said, you usually never get detected anyway.

Bypassing captcha automatically or with captcha bypass services where you pay per captcha completion?