News for hackers. Stuff that matters.

A comment on EU Open Web Search project kicked off. Parent comment

Re:EU Open Web Search project kicked off by jaimex2 on Tuesday September 20, 2022 @07:56PM (#32919351)
Wouldn't it be impossible to know if it ignored robots.txt?

Just because it crawled it doesn't mean it stored it.

[ Parent | on HN ]
Re:EU Open Web Search project kicked off by hrbf on Wednesday September 21, 2022 @04:48AM (#32922325)
Storage or not is entirely irrelevant to robots.txt directives. It guides automated access. It must be parsed first and excluded URLs must not be accessed at all.
[ Parent | on HN ]