News for hackers. Stuff that matters.

A comment on X updates its Terms to prohibit crawling/scraping of its data. Parent comment

Re:X updates its Terms to prohibit crawling/scraping of its ... by jraph on Friday September 08, 2023 @10:46AM (#37434173)
I wonder if search engines actually use rules for Googlebot when parsing some (specific?) robots.txt.

"Googlebot" might as well be the "Mozilla/5.0" of search index bot user agents.

[ Parent | on HN ]

There are no comments on this comment yet.