Re:U.S. FTC says Facebook misused privacy decree to shut dow...
by ClumsyPilot on Friday August 06, 2021 @09:04AM
(#28086579)
Hacking and other malicious behavious are actually illegal.
Either Crawling does not belong on that list, or google exects should be in jail.
Given that crawling is not malicious, what we are discussing now is 'someone is crawling my website in a way I dont like' which is a different gripe.
It mighthave some merit, but robots txt is not legally binding.
Re:U.S. FTC says Facebook misused privacy decree to shut dow...
by throwwebmaster on Friday August 06, 2021 @10:48AM
(#28087913)
As I stated before, Illegal or not, it is within the website owners’ rights to restrict access.
Nobody is legally forcing websites to allow access to everyone, and accordingly, nobody is altering the law by blocking access to people (crawlers, hackers, spammers, malcontents, or anybody really) that they feel are not welcome. So exercising one’s existing rights isn’t an act of making or altering laws.
I suggest reading up on what robots.txt is to further understand this.