Re:We should start to add “ai.txt” as we do for “robots.txt”
by User23 on Wednesday May 10, 2023 @02:06PM
(#35891050)
> there is no "legal" agreement to follow those rules
Yes there certainly is[1]. The robots.txt clearly specifies authorized use and violating it exceeds that authorization. Now granted good luck getting the FBI to doorkick their friends at Google and other politically connected tech companies, but as the law is written crawlers need to honor the site owner's robots.txt.
Yes there certainly is[1]. The robots.txt clearly specifies authorized use and violating it exceeds that authorization. Now granted good luck getting the FBI to doorkick their friends at Google and other politically connected tech companies, but as the law is written crawlers need to honor the site owner's robots.txt.
[1] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act