News for hackers. Stuff that matters.
Re:EU Open Web Search project kicked off by hrbf on Tuesday September 20, 2022 @03:40PM (#32916761)
I’ve already caught their crawler ignoring robots.txt directives on one of my sites, aggressively indexing explicitly excluded information.
Re:EU Open Web Search project kicked off by arjenpdevries on Tuesday September 20, 2022 @04:20PM (#32917255)
That cannot be true, as the project has yet to start. But anyone can start a crawler, so you may have encountered other people's software. We wouldn't be so unknowledgeable to ignore robots.txt ;-)
Re:EU Open Web Search project kicked off by hrbf on Wednesday September 21, 2022 @04:44AM (#32922294)
It was a crawler with the user agent "hgf AlphaXCrawl/0.1 (+https://www.fim.uni-passau.de/data-science/forschung/open-se...)", operated by the University Passau and Open Search Foundation named on your landing page. It would be a mighty big coincidence if this wasn't a project connected to this endeavor, especially when it confirms being an experimental crawler of said project at the UA URL.
Re:EU Open Web Search project kicked off by lizardactivist on Tuesday September 20, 2022 @04:04PM (#32917058)
Out of curiosity, what's the url for your website, and from what IP or host do their crawlers connect?
Re:EU Open Web Search project kicked off by hrbf on Wednesday September 21, 2022 @04:52AM (#32922355)
The main connecting IP was 195.113.175.41.
Re:EU Open Web Search project kicked off by jaimex2 on Tuesday September 20, 2022 @07:56PM (#32919351)
Wouldn't it be impossible to know if it ignored robots.txt?

Just because it crawled it doesn't mean it stored it.

Re:EU Open Web Search project kicked off by hrbf on Wednesday September 21, 2022 @04:48AM (#32922325)
Storage or not is entirely irrelevant to robots.txt directives. It guides automated access. It must be parsed first and excluded URLs must not be accessed at all.