News for hackers. Stuff that matters.

A comment on Show HN: robots.txt as a service, check web crawl rules through an API.

Re:Show HN: robots.txt as a service, check web crawl rules t... by dscpls on Sunday July 28, 2019 @05:30PM (#20549937)
Why a service and not a library?

It looks like a great way for you to discover URLs but like a terribly slow way for people to avoid implementing robots.txt rules.

[ Parent | on HN ]
Re:Show HN: robots.txt as a service, check web crawl rules t... by fooock on Monday July 29, 2019 @12:46PM (#20556573)
The project have multiple subprojects, and one of these is the parser. Any developer can compile or extend it without more effort and create a library. Just know to code in Kotlin / Java.

The aim of this project is only check if a given web resource can be crawled by a user-agent, but using a API

[ Parent | on HN ]