Re:Yahoo is making a return to search
by Nuzzerino on Monday January 30, 2023 @08:22PM
(#34589526)
That doesn’t change anything regarding the actual point of the comment.
[ Parent
| on HN ]
Your idea for a search competitor is to ignore robots.txt?
or an advertising competitor that ignores DNT!
oh wait
Re:Yahoo is making a return to search
by berkle4455 on Monday January 30, 2023 @10:12PM
(#34590386)
It's a just a text file.
This seems perfectly logical to some people:
Google ignores robots.txt: "Google is evil! They're trespassing on my webserver!"
Google follows robots.txt: "Google search results suck! They're not indexing GitHub!"