News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by lytedev on Friday March 26, 2021 @08:16PM (#26598534)
No, it's not. I can setup a login page and keep you out if I want. And I can do it however I want.
[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by DangitBobby on Saturday March 27, 2021 @09:32AM (#26602171)
But your login page will be public and subject to being crawled.
[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by henriquez on Saturday March 27, 2021 @12:44AM (#26599986)
My server, my rules.
[ Parent | on HN ]