I congratulate all of the authors whose work is included in this dataset on contributing their knowledge, skills, and perspective to humanity's various endeavors, both creative and technical. I hope that the fruits of their labors are returned to them, rather than being selfishly hoarded by the few with the resources necessary to produce those fruits, be they publishers, middlemen, or big tech.

Which is all to say that information shouldn't be hoarded and guarded. If it can produce something more than the sum of its parts we should use it to do so. The result of that should, on the same grounds, not be hoarded and guarded, doubly so being based on the work of others.

> I congratulate all of the authors whose work is included in this dataset on contributing their knowledge, skills, and perspective to humanity's various endeavours

Thank you. You know in some ways it's an honour and a privilege to live in such times of progress. The very act of publishing is to "let go", and hope that your words and ideas contribute to something bigger and beyond your life. I never believed much in "intellectual property" as it's all stuff that flows through us.

> I hope that the fruits of their labours are returned to them

They rarely are, because knowledge and creativity are not greatly valued in our time. But authors, artists and scientists go into that with eyes wide open these days. The rewards come in other ways, as the more you give and put into life the more you get out.

> rather than being selfishly hoarded by the few with the resources necessary to produce those fruits

This is not what we fear. Hoard away. We will simply take back what is ours, whenever we desire it. The hoarders will never win against what they call "piracy", because they have no moral right. In the long run, they are on the wrong side of history.

Far worse, and more likely is that the creative and technical works of generations of artists and scientists are going to be turned to exactly the opposite of what they would want. They will be used to harm and disempower humans, divide society instead of heal it, and even make the pursuits of art, science and knowledge irrelevant.

We cannot take back our words, or our formulas, or our paintings or our songs. But we can take back tech.

It will produce "something more" for the already-wealthy who control the technology. For instance, LLMs will eliminate the need for some customer service jobs, increasing the profit margin for the existing executives and shareholders, while eliminating entry-level jobs from the job market.

Call me an idealist but I don't think humans should be spending their time on jobs a computer can do.

The solution to wealth disparity cannot include "invent menial untalented high-paying labor for people to do".

Yeah why should humans do bothersome labor like...creating literature?

You're romanticising the creation of literature for some reason.

There's no difference between a machine replacing a human writing a book and a machine replacing a human making a piece of wooden furniture. Literature and craftwork are equally rewarding and beneficial imo.

And people still make wooden furniture by hand even after IKEA. If someone wants cheap and uninteresting they'll buy IKEA, if someone wants an interesting and unique piece, or to support handmade things then they'll buy from a woodworker.

That sounds nice except that it didn't happen. They scraped this off many sites whose authors published the material in ways that wouldn't legally allow that. They often publish for a mix of self interest and public benefit. The more you look, the more you find that much altruism is actually self-interest at work. Some have specific terms, too. Let's look at examples in The Pile.

It might be something that's part of their job, requires citations since they value credit, optionally bans commercial use, and maybe has a patent. Arxiv papers are a mix of that. Many on StackOverflow and Hacker News want attribution with some asserting copyright in their comments. Film producers usually want the subtitles to accompany sales of their movies. For FreeLaw, the material is public by law about people who might have never even wanted to testify or be remembered. FreeLaw itself was seeking donations for its service with an additional request to protect the names of people in the dataset. StackOverflow's license explicitly bans copying their data without permission with many individual users also wanting self-promotion to happen side by side with their answers.

So, many things that are in The Pile are works people published hoping to gain some benefit in return. They often had terms that banned their reproduction in ways that prevented them from getting that benefit. They were just public for humans to read and learn from. Some allowed sharing but just wanted credit.

The A.I. users of these works ignore all of that by taking what they made conditionally available without meeting the conditions that benefit the authors. Whereas, if you got the same content on Hacker News or Arxiv, the authors might benefit from it. Even a pirate would benefit them more than A.I. companies because the users would often at least know the author or source site. So, the fruits of their labor were taken, not given, by those who are the least beneficial to them.

I will note that some people do publish truly free content that has no strings attached. Mostly public domain or CC-0. Those are exceptions that might fit your description.

I specifically did not say that they had contributed willingly. They have contributed nonetheless.

I doubt that anyone is going to download and search through over 800 TB just to find a badly formatted copy of some book that could be found much quicker on different websites with better formatting. Authors are losing fractional cents here at most.

so just like Office Space? (paraphrasing) "We steal a fraction of a cent from each transaction, who do we hurt? Nobody. We just put the remainder into our account!"

Sorry that's not how damages are calculated in the US tort system.

I do not know how damages are calculated in the US tort system. What do they say about the books3 dataset?

I also think that the case is different here, since in your example, there is a specific amount of money being stolen, while in the books3 case, there is an unspecified amount of money not being made by the authors.

I am pretty sure if the authors were trying to license their works for this purpose we would just not use them at all; it is difficult to see under what circumstances they would stand to profit from this other than by suing people after the fact over it.

I think you could argue that authors could profit from their works being cited in an LLM response. It could drive sales of their works much like citations do on the web. The counter argument is that and LLM could give you the Clif Notes version of the work and thus taking away a portion of sales.

In a world where the options were to

1) pay the author,

2) implement guaranteed citation of the author any time the model gave an answer that was directly derivative, with an option to not do so if the summary was sufficiently vague, or

3) ignore the author's book completely as training data

we would all choose 3).

And the authors would probably be very happy that you did.

The penalty is up to $150k per violation.

For uploading, not downloading.

they have stole nothing, they make no profit from it as well.

Oh, so there aren't AI companies charging for access to private models?

Who are they? Why do you mix up the guys who prepared the data with the other guys who used this data and making money from a vague memory of that data?