Re:U.S. FTC says Facebook misused privacy decree to shut dow...
by throwwebmaster on Friday August 06, 2021 @10:48AM
(#28087913)
As I stated before, Illegal or not, it is within the website owners’ rights to restrict access.
Either Crawling does not belong on that list, or google exects should be in jail.
Given that crawling is not malicious, what we are discussing now is 'someone is crawling my website in a way I dont like' which is a different gripe.
It mighthave some merit, but robots txt is not legally binding.