Re:Court dismisses Genius lawsuit over lyrics-scraping by Go...
by Polylactic_acid on Wednesday August 12, 2020 @01:26AM
(#24128833)
robots.txt is designed to keep garbage off search results. It has absolutely no power to prevent a bot to do anything. Also if the site added robots.txt they might as well shut down because their entire userbase comes from people searching lyrics on google.
[ Parent
| on HN ]
Re:Court dismisses Genius lawsuit over lyrics-scraping by Go...
by mikemotherwell on Wednesday August 12, 2020 @02:56AM
(#24129272)
Other way around. It was invented to stop crawling. Indexing is still technically allowed even when blocked by robots.txt from crawling.