News for hackers. Stuff that matters.

A comment on OpenAI Announces SearchGPT. Parent comment

Re:OpenAI Announces SearchGPT by wilg on Thursday July 25, 2024 @02:56PM (#41072095)
If I were making a search engine or AI crawler, I would simply pose as Googlebot
[ Parent | on HN ]
Re:OpenAI Announces SearchGPT by jsheard on Thursday July 25, 2024 @02:58PM (#41072126)
Google actually provides means of validating whether a request really came from them, so masquerading as Googlebot would probably backfire on you. I would expect the big CDNs to flag your IP address as malicious if you fail that check.

https://developers.google.com/search/docs/crawling-indexing/...

[ Parent | on HN ]
Re:OpenAI Announces SearchGPT by wilg on Thursday July 25, 2024 @03:04PM (#41072197)
You could maybe still only follow robots.txt rules for Googlebot.
[ Parent | on HN ]