News for hackers. Stuff that matters.

A comment on Cloudflare.com's Robots.txt.

Re:Cloudflare.com's Robots.txt by jsheard on Sunday November 17, 2024 @08:27AM (#42164090)
This is what happens if your robot isn't nice

  > curl -I -H "User-Agent: Googlebot" https://www.cloudflare.com
  HTTP/2 403
[ Parent | on HN ]
Re:Cloudflare.com's Robots.txt by jamesog on Sunday November 17, 2024 @09:02AM (#42164220)
That's not from robots.txt, but their Bot Management feature which blocks things calling themselves Googlebot that don't come from known Google IPs.
[ Parent | on HN ]
Re:Cloudflare.com's Robots.txt by speedgoose on Sunday November 17, 2024 @10:21AM (#42164616)
Are GCP IPs considered Google IPs?
[ Parent | on HN ]
Re:Cloudflare.com's Robots.txt by jgrahamc on Sunday November 17, 2024 @10:27AM (#42164657)
No.
[ Parent | on HN ]
Re:Cloudflare.com's Robots.txt by crop_rotation on Sunday November 17, 2024 @10:25AM (#42164648)
No I am very sure they are not.
[ Parent | on HN ]
Re:Cloudflare.com's Robots.txt by judge2020 on Sunday November 17, 2024 @12:58PM (#42165651)
For reference https://developers.google.com/search/docs/crawling-indexing/...
[ Parent | on HN ]