News for hackers. Stuff that matters.

A comment on Anyone got a contact at OpenAI. They have a spider problem.

Re:Anyone got a contact at OpenAI. They have a spider problem by 1317 on Thursday April 11, 2024 @05:02PM (#40006770)
He's not done his robots.txt properly, he's commented out the bit that actually disallows it

  # silly bing
  #User-agent: Amazonbot          
  #Disallow: /

  # buzz off
  #User-agent: GPTBot
  #Disallow: /

  # Don't Allow everyone
  User-agent: *
  Disallow: /archive

  # slow down, dudes
  #Crawl-delay: 60
[ Parent | on HN ]
Re:Anyone got a contact at OpenAI. They have a spider problem by haeffin on Thursday April 11, 2024 @05:18PM (#40006924)
The contents changed between then and now.
[ Parent | on HN ]