Re:GPTBot – OpenAI’s Web Crawler
by kevingadd on Monday August 07, 2023 @01:12PM
(#37037410)
Keeping you honest with incognito crawling is something they have to do anyway, to catch various tricks and scams - malware served up to users, etc.
[ Parent
| on HN ]
So robots.txt is meaningless if they have to violate it to check for malicious content in blocked off pages anyways.
Re:GPTBot – OpenAI’s Web Crawler
by thenonsequitur on Monday August 07, 2023 @09:34PM
(#37043362)
Well if you are blocking access to their crawler, I'd imagine they'd have no need to use an incognito crawler to check for malicious content. Why would they care if that content is not ending up in their index anyway?
Presumably, the incognito crawlers are only used on sites that have already granted the regular crawler access. That's content that ends up in their index which they want to vet.