News for hackers. Stuff that matters.

A comment on BBC threatens AI firm with legal action over unauthorised content use. Parent comment

Re:BBC threatens AI firm with legal action over unauthorised... by hadrien01 on Friday June 20, 2025 @10:25AM (#44328024)
They also write this:

> Since a user requested the fetch, this fetcher generally ignores robots.txt rules.

[ Parent | on HN ]
Re:BBC threatens AI firm with legal action over unauthorised... by simonw on Friday June 20, 2025 @10:54AM (#44328294)
Oh wow, I missed that! That's from the docs for that Perplexity‑User user-agent, at which point presumably there's no point in listing that in robots.txt at all?
[ Parent | on HN ]
Re:BBC threatens AI firm with legal action over unauthorised... by dabeeeenster on Friday June 20, 2025 @10:32AM (#44328085)
I mean, that's just not true.
[ Parent | on HN ]
Re:BBC threatens AI firm with legal action over unauthorised... by esskay on Friday June 20, 2025 @12:17PM (#44329224)
Which part? It's widely established and known that many AI crawlers are ignoring the robots.txt file, perplexity being one of them [1]

[1]https://www.tomshardware.com/tech-industry/artificial-intell...

[ Parent | on HN ]
Re:BBC threatens AI firm with legal action over unauthorised... by bitpush on Friday June 20, 2025 @01:27PM (#44329915)
> Since a user requested the fetch, this fetcher generally ignores robots.txt rules.

Normally the expecation is that the user-agent faithfully presents the content it fetched.

If I make a browser that fetches bbc.com, and strips away ads and presented it to users - I would expect BBC to not like it and block the user-agent from accessing it. It isnt a robots.txt thing. It is a user-agent thing.

[ Parent | on HN ]