News for hackers. Stuff that matters.
Re:Brave Search launches own image and video search by cvalka on Thursday August 03, 2023 @06:43PM (#36993092)
They are right and you are wrong. If some web page is publicly available, it should be indexed. Scraping neutrality, please.
Re:Brave Search launches own image and video search by tympious on Thursday August 03, 2023 @08:07PM (#36993785)
What if I want what I publish to be known only by word of mouth?

What if I consider (some or any of) my ideas to be un-indexable, not directly suitable to representation in any hierarchy other than those I may set them in?

Re:Brave Search launches own image and video search by Grimburger on Thursday August 03, 2023 @08:16PM (#36993846)
Then you should hide them behind a url that isn't linked elsewhere on your site that you can easily propagate by word of mouth only.

    example.com/correcthorsebatterystaple
If you consider "word of mouth" to be public posts on a forum which millions can read at any time then block googlebot IP's
Re:Brave Search launches own image and video search by tympious on Thursday August 03, 2023 @09:27PM (#36994302)
Yes, sorry, it was a rhetorical question in response to previous.

Taking the obvious steps you suggest (along with robots.txt or eqiv.), it would seem fair to expect that Brave, Bing, whomever, would not feel it their neutral/natural domain to include in a public index.

Re:Brave Search launches own image and video search by jaystraw on Friday August 04, 2023 @04:16AM (#36996984)
...after decades of what i considered friendship, here you are on hn talking about my horse battery staple
Re:Brave Search launches own image and video search by memefrog on Friday August 04, 2023 @08:20AM (#36998949)
Then dont publish it.
Re:Brave Search launches own image and video search by waithuh on Friday August 04, 2023 @08:33AM (#36999087)
Heavily disagree. I own the server, thus the website. I should be able to allow or disallow any type of web crawler/scraper i want. Similar to how you cant easily regulate whats in a website without lawsuits and takedowns, you cant regulate how discoverable a website is.
Re:Brave Search launches own image and video search by ric2b on Friday August 04, 2023 @01:35PM (#37003193)
> I should be able to allow or disallow any type of web crawler/scraper i want.

You're certainly allowed to try, but I don't see why indexers should be mandated to collaborate with you. They serve their users, not you.

Re:Brave Search launches own image and video search by tympious on Friday August 04, 2023 @10:13PM (#37008301)
Will their users appreciate that they disregard the intent of the authors of what they index?

I mean, "allow" or "regulate" don't _really_ apply here - there was never any enforcement regime around robots.txt, just a convention based on the general expectation that you don't claim ownership of whatever passes your line of sight.