News for hackers. Stuff that matters.
Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by threeseed on Friday February 07, 2025 @07:33AM (#42971846)
> Google itself got big by indexing other people's data without compensation

Wrong.

a) Robots.txt which defines what content you wish to make available to third parties predates every search engine including Google. Web site owners chose to make it available to Google and search engines have respected their wishes despite it not being in their best interest.

b) The difference here is that OpenAI, Meta etc have not even tried to honour the wishes of copyright holders. They just considered everything as theirs.

c) Google grew big because it had no ads, fast interface and PageRank was significantly better. It wasn't because it had the most comprehensive index.

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by karamanolev on Friday February 07, 2025 @07:35AM (#42971867)
> Web site owners chose to make it available to Google.

Strong disagree. Since robots.txt is optional and the default is "crawl me as you please", website owners don't "choose to make it available", they just don't choose to make it non-available.

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by XorNot on Friday February 07, 2025 @07:53AM (#42972005)
That's a functionally meaningless distinction. If you setup a web server that responds to requests, then you're choosing to make content available because your server can choose to not respond to requests. The entire protocol includes mechanisms to negotiate access.
Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by eviks on Friday February 07, 2025 @08:16AM (#42972167)
This is a meaningless simplification. In this framework "robots.txt" has no role, because your server "can choose" not to respond. Heck, even DDOS is fine, because "protocol"
Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by jokethrowaway on Friday February 07, 2025 @08:16AM (#42972157)
Granting access and granting right to redistribute (even just title + snippet) and use your content commercially are two completely different things.
Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by XorNot on Friday February 07, 2025 @03:29PM (#42977111)
And yet it is legal to produce and redistribute summaries as sufficiently transformative derivative works, and this has been court tested[1]. Of course in Australia we passed rather specific laws to the contrary, because lo and behold Rupert Murdoch wanted money and gosh darn it our government was going to give it to him[2].

[1] https://www.practicalecommerce.com/Search-Engines-Indexing-a...

[2] https://www.alrc.gov.au/publication/copyright-and-the-digita...

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by tobyhinloopen on Friday February 07, 2025 @07:52AM (#42971998)
a) If you don't have a robots.txt, you're indexed by default. It's opt-out, not opt-in. If you do nothing, you're being indexed.
Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by antiframe on Friday February 07, 2025 @12:50PM (#42975418)
It's an opt-out of an opt-in. If you run a webserver hosting your files, you already opted-in to people accessing that data. If you then don't go ahead an configure it properly, that's not exactly "opt-out" anymore. By default your files are not accessible to the network, you have to first opt-in to serving them.
Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by tobyhinloopen on Saturday February 08, 2025 @03:44AM (#42981510)
Google makes a copy of your data and serves that data to users before they visit your site.

Also google cache allows users to get a copy of your site without visiting your site.

Why can they republish your data while we cannot? Why do we have to opt-out?

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by RALaBarge on Friday February 07, 2025 @07:35AM (#42971861)
To your first point, the op said without compensation, not without permission.
Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by boesboes on Friday February 07, 2025 @07:36AM (#42971869)
Wrong. Google ignores robots.txt entirely
Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by threeseed on Friday February 07, 2025 @07:41AM (#42971925)
I wasn't aware. Can you please update Wikipedia then: https://en.wikipedia.org/wiki/Robots.txt

Maybe also get Google to update their docs: https://developers.google.com/search/docs/crawling-indexing/...

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by phit_ on Friday February 07, 2025 @08:08AM (#42972111)
their own docs also specify that the robots.txt does not stop indexing or showing up in search, they even bolded it "it is not a mechanism for keeping a web page out of Google"

https://developers.google.com/search/docs/crawling-indexing/...

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by alphan0n on Friday February 07, 2025 @08:48AM (#42972454)
The only way for links to appear in a Google search would be to host a public resource, that is linked from another public resource.

If you have specified in your robots.txt that you do not want the page(s) or directories ingested then only the url is indexed (if it is linked from another page). It does prevent the public display of the content of a page and creation description/summary.

https://support.google.com/webmasters/answer/7489871?hl=en

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by threeseed on Friday February 07, 2025 @03:55PM (#42977373)
From the docs:

“While Google won't crawl or index the content blocked by a robots.txt file”

They will show the URL if someone else has linked to it. But the content itself is not indexed.

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by nottorp on Friday February 07, 2025 @07:50AM (#42971988)
It must be nice to believe everything people say by default... ;)
Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by fredgrott on Friday February 07, 2025 @07:35AM (#42971866)
point c is wrong...they had ads since the original yahoo contract....
Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by threeseed on Friday February 07, 2025 @07:38AM (#42971901)
Yahoo contract was 2 years after it launched.

I remember using Google the day it went public and it had no ads which made it unique compared to Altavista.

Re:Meta torrented & seeded 81.7 TB dataset containing copyri... by veggieroll on Friday February 07, 2025 @08:26AM (#42972258)
Robots.txt is irrelevant after hiQ Labs v. LinkedIn (2019)