News for hackers. Stuff that matters.
Re:Googlebot’s JavaScript random() function is deterministic by amelius on Wednesday February 07, 2018 @04:43AM (#16323311)
Nice to see fingerprinting used against Google (instead of by Google). But this is easy to fix, so I don't expect this to work for much longer.
Re:Googlebot’s JavaScript random() function is deterministic by jrochkind1 on Wednesday February 07, 2018 @09:29AM (#16324579)
I think how easy it is for Google to fix to make detecting googlebot this way harder depends on why Googlebot is doing it in the first place, which we don't really know. If it's done for performance reasons, or for predictability reasons (rendering the same page twice guaranteed or at least more likely to produce the same result), it might be difficult to change without cost.

But I believe Googlebot always faithfully sends it's user-agent. Is there a reason Google would care about 'fixing' this to make Googlebot harder to detect via random() predictability, when you can always just detect it via user-agent anyway? I'm not sure, curious if others have thoughts!

Re:Googlebot’s JavaScript random() function is deterministic by Kesty on Wednesday February 07, 2018 @12:02PM (#16325927)
Google has checks in place to see if someone is serving things to GoogleBot differently that the rest of the users. So it almost definitely has bots that double checks pages without the user-agent.

If the "disguised" googlebot is the same as the actual one, chances are it is since it would want to be as close as possible to not flag false positives, and use the same seed for consistency then you might be able to use that to avoid detection on the fact that you are serving google something different than normal users.

Newspaper used/do that to be able to have their full article content indexed while serving a paywall to everyone else.

Re:Googlebot’s JavaScript random() function is deterministic by codedokode on Wednesday February 07, 2018 @06:07AM (#16323666)
It is much easier to detect Googlebot by IP address or User-Agent though.
Re:Googlebot’s JavaScript random() function is deterministic by p49k on Wednesday February 07, 2018 @06:17AM (#16323703)
I’d assume that they occasionally send requests with different IPs and user-agents to detect cloaking.
Re:Googlebot’s JavaScript random() function is deterministic by leni536 on Wednesday February 07, 2018 @06:59AM (#16323837)
Maybe they change the behavior of their random() function as well. The deterministic random() function can very well be a red herring.
Re:Googlebot’s JavaScript random() function is deterministic by lucideer on Wednesday February 07, 2018 @07:07AM (#16323874)
I'd guess if they're crawling purely for the purpose of detecting cloaking (which would be a much smaller-scale job than the standard Googlebot indexing), they'd just use Chrome Headless[0][1]

[0] https://intoli.com/blog/not-possible-to-block-chrome-headles...

[1] http://antoinevastel.github.io/bot%20detection/2018/01/17/de...

Re:Googlebot’s JavaScript random() function is deterministic by thecatspaw on Wednesday February 07, 2018 @08:03AM (#16324089)
I doubt that they use chrome at all. They probably just fetch the page over http and let it run in a JS sandbox. hence the deterministic random function (which I assume is not there in chrome?)
Re:Googlebot’s JavaScript random() function is deterministic by DamonHD on Wednesday February 07, 2018 @08:51AM (#16324350)
2018/01/18: Chrome 41: The Key to Successful Website Rendering: "Google updated their Search Guides and announced that they use Chrome 41 for rendering."

https://www.elephate.com/blog/chrome-41-key-to-website-rende...

Re:Googlebot’s JavaScript random() function is deterministic by codedokode on Thursday February 08, 2018 @01:09AM (#16330544)
It is interesting that when I use Chromium 46 (which is newer than Googlebot) I get warnings from Github and Google Docs about an outdated or unsupported browser, and I get plain HTML Github pages without JS. But Google uses even older browser than I. So even Google cannot cope with browser version race and keep their browsers updated although they require this from the users.
Re:Googlebot’s JavaScript random() function is deterministic by lucideer on Thursday February 08, 2018 @09:15AM (#16332122)
According to that article, Googlebot is "based on Chromium 41", it doesn't actually use Chromium 41 directly, it's still a distinct, separately developed browser.

The warning you're seeing is

a) probably shown based on UA, and Googlebot has a very different UA to the Chromium browser

b) warning users who will need to browse/actively use the site. Googlebot simply parses content, so has no use for quite a lot of the active functionality on the site. As such, it will generally not need to support all of the features used by that active functionality, just barely enough to be able to read content.

Re:Googlebot’s JavaScript random() function is deterministic by lucideer on Wednesday February 07, 2018 @09:38AM (#16324637)
I'm sure they don't use Chrome for Googlebot, that would be quite inefficient. My comment was about cloaking detection.
Re:Googlebot’s JavaScript random() function is deterministic by taneq on Wednesday February 07, 2018 @07:42AM (#16324007)
Maybe that's just what they want you to think! /tinfoil
Re:Googlebot’s JavaScript random() function is deterministic by chrisper on Wednesday February 07, 2018 @06:31AM (#16323749)
Yes, but their IPs are registered to Google Inc.
Re:Googlebot’s JavaScript random() function is deterministic by TeMPOraL on Wednesday February 07, 2018 @08:09AM (#16324114)
Are they?
Re:Googlebot’s JavaScript random() function is deterministic by joombaga on Wednesday February 07, 2018 @09:56AM (#16324754)
Yes. Are you implying they are not?
Re:Googlebot’s JavaScript random() function is deterministic by TeMPOraL on Wednesday February 07, 2018 @10:59AM (#16325295)
Yes? I mean, scanning for cloaking only from their own IP range sounds ridiculous.
Re:Googlebot’s JavaScript random() function is deterministic by codedokode on Wednesday February 07, 2018 @06:33AM (#16323757)
I think that cloaking is very difficult to detect automatically without using humans. You have to distinguish for example between a paywall and real cloaking.
Re:Googlebot’s JavaScript random() function is deterministic by xstartup on Wednesday February 07, 2018 @04:57AM (#16323372)
I don't think it's particularly easy to fix without fixing a lot of other things. Let's see how much time it takes them. :)
Re:Googlebot’s JavaScript random() function is deterministic by sergz on Wednesday February 07, 2018 @05:03AM (#16323397)
Perhaps for performance or testing reasons they compiled V8 with a predefined seed, keywords are "V8", "mksnapshot", "--random-seed". It should not be too difficult to fix it if there are no artificial restrictions by use cases.