News for hackers. Stuff that matters.

A comment on Setting User-Agent Field? (1996). Parent comment

Re:Setting User-Agent Field? (1996) by rbinv on Thursday January 07, 2021 @08:56AM (#25671034)
Today, Google even crawls, renders and indexes URLs sent from Chrome browsers via dial-up IPs using regular browser User-agents, so... no. Other than that, Googlebot seems to obey most of the time.

I don't blame them. Cloaking couldn't be detected otherwise.

[ Parent | on HN ]
Re:Setting User-Agent Field? (1996) by toper-centage on Thursday January 07, 2021 @03:17PM (#25676773)
I think i read somewhere that Google crawlers sometimes pretend to be real browsers to prevent you from displaying a different page the crawler. So, no.
[ Parent | on HN ]