No it's the death of the corporate content hosting web - the open web was never about making money with your blog post/irc chat/usenet group/etc content, at least in my opinion.

Let data be free! If someone wants to use it to make money, well, it's open, just like open source. It's still not okay to take open source work and claim it as your own, which is what copyright should be limited to. Stealing a photo or plagiarizing an essay is intrinsically different than just having a copy read by something, be it human or an mechanical process such as training a LLM.

Maybe I don’t want my non-corporate art to be a part of some large corporation’s training data.

User-agent: GPTBot Disallow: /

ATMO you shouldn't have to maintain knowledge of what kind of crawler bot exist and having to maintain deny list. It should be the opposite, only expressedly allowed content should be crawled by mainaining allow lists.

You can do the opposite since the inception of robots.txt: User-agent: * Disallow: / and then whitelist google bot and whatnot. Most of the web is already configured this way. Just check robots.txt of any major website, e.g. https://twitter.com/robots.txt

The Allow: directive was an extension to robots.txt added later.

That was my gut reaction too, but presumably unless it becomes regulated, at least some competitors to OpenAI won't respect any robots.txt and thus any open content might be training data.

User-Agent: <new technology category>Bot

Once again I have to point out that LLMs are not and will not be restricted to "large corporations". Even if initial training is prohibitively expensive, open base models already exist.

"corporate this" and "corporate that" are handy rhetorical flourishes but they distort the debate.

if you want to give away your data for free, do it, but speak for yourself, not everyone is in position to be able to do it. Just like you can't force other people to give away their software (or anything else, really) for free if they choose to charge for it.

[delayed]

Let's assume that happens. How do I hedge against it? Is there a convenient way to mirror the bits of the web that are open now? Perhaps a mirror of archive/WayBack machine that could be viewed locally similar to Wikipedia dumps?

Yes, commoncrawl indexes

100TB. Dang! I will need to expand my storage significantly to mirror it.

100TB so far ;)

And that's probably just 1 year. There's data that's vanished over time. The point of all this is that they're point-in-time references of the internet.

the more practical thing is that (the royal) they have set up a rclone like utility that allows you to retrieve a slice of the whole snapshot, if you know your domain.

it would be extremely funni if commoncrawl was a de-facto ipfs, allowing queries along domains (or arbitrary dimensions/clustering) of interest.

Really? Seems like it's not so different conceptually to search engines. They also make money by indexing "training data" of a sort. But the open web trundles on regardless, because the search engines found ways to cut the content producers in on it. I see no reason why that can't be the case here too, with AI companies training their models to act more like search engines when data comes from certain sources - i.e. they'll point you in the right direction but not directly answer you. That will suck for the AI users, but, for many questions the right answer will be available in open or bulk licensable training sets anyway. For example you can get legal access to nearly all books by doing deals with publishers, as Google Books has demonstrated, you can get access to map data by generating it yourself or doing deals with digital map companies etc.

>Really? Seems like it's not so different conceptually to search engines. They also make money by indexing "training data" of a sort.

OK, so if a writer X has a blog to put up samples of their work to drive people to buy books and to get writing assignments and someone uses ChatGPT to write something in the style of X - this naively seems like a hit on that author's ability to sell their skills.

And I don't think it is fixable by making the AI act more like a search engine.

Seems like a rerun of the argument over snippets, or the "answer onebox" as Google used to call it where the info you need is directly inlined in the SERP rather than being behind a link.

it is exactly a rerun of that argument, except that the UX of AI is different.

Basically AI is structured to make every result an answer onebox whereas in search this only happens sometimes.

Yes I would say that is very similar. Many sites saw their revenue crater, based on data they themselves had created.

> this naively seems like a hit on that author's ability to sell their skills

If the AI is good, then the author's economic outlook is bad regardless of style.

Regardless of if the AI is or isn't good, then I can still see it being brand damaging, but my gut feeling (IANAL) is that this is more of a trademark issue than a copyright issue, as it's passing off as something it isn't.

It would work both ways. I could also imagine someone asking: “what books should I read to learn about X?” And the LLM could drive sales toward that author’s books.

It’s not clear to me that having their content “unindexed” is good for authors. It’s probably good and bad at the same time.

yes indeed, search engines settled that problem a long time ago in a civilized way. The more of these lawsuits, the harder the wild west of "training" will have to search for an acceptable solution to what is currently web-scale theft of data.

Is this open web in the room with us? Cheekiness aside, the myth of beautiful open web always seems to be exagerated. Most content on the web is already generated, ugly and spammy. Most of the traffic is already owned by mega corporations.

One could easily argue that it's unlikely it'll get worse - if anything, AI could empower competition as now a group of 3 passionate, free writers can compete with agenda-driven, for profit corporations on a similar level. This could very well make the web more open and free as it makes the web more accessible.

> Most content on the web is already generated, ugly and spammy.

Now? Yes. I grew up when it was just ugly, and by ugly I mean animated gif backgrounds with obvious seams on the tile boundary.

*old man shakes fist at The Cloud*