News for hackers. Stuff that matters.

A comment on GPTBot – OpenAI’s Web Crawler. Parent comment

Re:GPTBot – OpenAI’s Web Crawler by dmitrysergeyev on Monday August 07, 2023 @04:54AM (#37032008)
They could in theory combat it by comparing results with a second crawler that uses a different User Agent.
[ Parent | on HN ]
Re:GPTBot – OpenAI’s Web Crawler by mdaniel on Monday August 07, 2023 @11:56AM (#37036168)
If they were going to the amount of energy to do a 2nd crawl using a different user agent, then why bother advertising the user agent at all and just feed it the Chrome one like every other home-grown spider does
[ Parent | on HN ]