Re:GPTBot – OpenAI’s Web Crawler
by yosito on Monday August 07, 2023 @03:09AM
(#37031223)
Is there any argument in favor of commercial websites allowing GPTBot to crawl them? It's not like Google where allowing crawling brings you traffic. In fact, it's pretty much the opposite.
[ Parent
| on HN ]
I assume by "commercial websites" you mean specifically "websites whose whole purpose is to have information in the website that you view in return for running ads?" Generally speaking, if I have a website where I have information about my business, then most likely it benefits me for people's LLMs to know that information, for the same reason I might buy an ad for my business.
Think about an AI as a personal assistant - for the whole world. Would you want the assistant to know about your business?
In most cases I do think so. It could mention you in a conversation, the analog to you appearing in Google Search results. And maybe even better, provide the necessary context to generate more real customers for your business. You don’t want traffic to your website, you want customers to your business. If you currently convert 10% of your traffic to customers, you’d be happy with 10% of the traffic of which you convert all to customers, because they are already converted before they even clicked your link.
Re:GPTBot – OpenAI’s Web Crawler
by terrytanys on Wednesday August 09, 2023 @08:09PM
(#37070116)
OpenAI have to know they are going into consumer business not just an AI research company.