News for hackers. Stuff that matters.
Re:The New York Times Launches a Strong Case Against Microso... by quonn on Sunday January 07, 2024 @10:13AM (#38901825)
There is a robots.txt. If this mechanism would not exist, I would argue that the publishers have a case regarding Google (they are trying). But it does exist and so they shouldn‘t.
Re:The New York Times Launches a Strong Case Against Microso... by mrkramer on Sunday January 07, 2024 @10:58AM (#38902257)
Somewhere I read that a lot of websites don't even have robots.txt so I wonder does Google's crawler bots skip those websites or do they crawl them as someone said as a "fair use".

Speaking generally about internet search engines; majority of the websites on the web want to get discovered and attract people to their whatever (web store, web community, web blog etc.)

The interesting idea I was thinking about is that a big company which operates a search engine like a Microsoft (Bing), can pay for example popular websites like Reddit, NYT, WSJ to have exclusive crawling right and therefore block Google from crawling their websites and only allow Bing to crawl them. This would then spark search engine content war which could significantly weaken Google's monopoly because a lot of people would switch to Microsoft Bing only because Reddit results show up in their search queries. This would be akin to Netflix investing billions of dollars in the exclusive content which then brings them a lot of new subscribers and lot of new revenue. In another words - user acquisition.

Re:The New York Times Launches a Strong Case Against Microso... by figassis on Sunday January 07, 2024 @10:45AM (#38902147)
Missing robots.txt does not automatically grant you the right to use copyrighted content.
Re:The New York Times Launches a Strong Case Against Microso... by cxr on Sunday January 07, 2024 @11:25AM (#38902483)