News for hackers. Stuff that matters.
Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by threeseed on Friday February 07, 2025 @07:41AM (#42971925)
I wasn't aware. Can you please update Wikipedia then: https://en.wikipedia.org/wiki/Robots.txt

Maybe also get Google to update their docs: https://developers.google.com/search/docs/crawling-indexing/...

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by phit_ on Friday February 07, 2025 @08:08AM (#42972111)
their own docs also specify that the robots.txt does not stop indexing or showing up in search, they even bolded it "it is not a mechanism for keeping a web page out of Google"

https://developers.google.com/search/docs/crawling-indexing/...

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by alphan0n on Friday February 07, 2025 @08:48AM (#42972454)
The only way for links to appear in a Google search would be to host a public resource, that is linked from another public resource.

If you have specified in your robots.txt that you do not want the page(s) or directories ingested then only the url is indexed (if it is linked from another page). It does prevent the public display of the content of a page and creation description/summary.

https://support.google.com/webmasters/answer/7489871?hl=en

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by threeseed on Friday February 07, 2025 @03:55PM (#42977373)
From the docs:

“While Google won't crawl or index the content blocked by a robots.txt file”

They will show the URL if someone else has linked to it. But the content itself is not indexed.

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by nottorp on Friday February 07, 2025 @07:50AM (#42971988)
It must be nice to believe everything people say by default... ;)