News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by d110af5ccf on Friday March 26, 2021 @07:07PM (#26598076)
> Isn't that the website owners right though?

No. The internet is public. Publishers shouldn't get any say in who accesses their content or how they do it. As far as I'm concerned, the fact that they do is a bug.

[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by lytedev on Friday March 26, 2021 @08:16PM (#26598534)
No, it's not. I can setup a login page and keep you out if I want. And I can do it however I want.
[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by DangitBobby on Saturday March 27, 2021 @09:32AM (#26602171)
But your login page will be public and subject to being crawled.
[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by henriquez on Saturday March 27, 2021 @12:44AM (#26599986)
My server, my rules.
[ Parent | on HN ]