Re:Brave Search launches own image and video search
by waithuh on Friday August 04, 2023 @08:33AM
(#36999087)
Heavily disagree. I own the server, thus the website. I should be able to allow or disallow any type of web crawler/scraper i want. Similar to how you cant easily regulate whats in a website without lawsuits and takedowns, you cant regulate how discoverable a website is.
[ Parent
| on HN ]
Re:Brave Search launches own image and video search
by ric2b on Friday August 04, 2023 @01:35PM
(#37003193)
> I should be able to allow or disallow any type of web crawler/scraper i want.
You're certainly allowed to try, but I don't see why indexers should be mandated to collaborate with you. They serve their users, not you.
Re:Brave Search launches own image and video search
by tympious on Friday August 04, 2023 @10:13PM
(#37008301)
Will their users appreciate that they disregard the intent of the authors of what they index?
I mean, "allow" or "regulate" don't _really_ apply here - there was never any enforcement regime around robots.txt, just a convention based on the general expectation that you don't claim ownership of whatever passes your line of sight.