OpenAI crawler burning money for nothing

Re:OpenAI crawler burning money for nothing by gbertb on Sunday January 19, 2025 @02:34PM (#42760823)

how are the links structured in the ahref tag? is it relative or absolute? if relative, then thats prob why.

[ Parent | on HN ]

Re:OpenAI crawler burning money for nothing by babuskov on Sunday January 19, 2025 @03:44PM (#42761605)

Relative.

For example, the page:

https://website/blog/1-post

contains:

href="2-post"

Browsers and other bots like Google Bot correctly interpret this as a link to

https://website/blog/2-post

While OpenAI crawler goes to:

https://website/blog/1-post/2-post

I wonder is there some way to report this bug to them?

[ Parent | on HN ]

Re:OpenAI crawler burning money for nothing by fullstackwife on Sunday January 19, 2025 @06:01PM (#42763007)

Google recommends absolute urls:

https://web.archive.org/web/20221208150134/https://www.webma...

[ Parent | on HN ]

Re:OpenAI crawler burning money for nothing by AznHisoka on Sunday January 19, 2025 @06:55PM (#42763422)

I actually think OpenAI is right, unless you have a base url tag? That’s a relative url and its relative to the current url you are on, not the root domain

[ Parent | on HN ]