That's a lot of assumptions about human level intelligence. We do know that the more you read, the better you can write, and a lot of our text is "inspired" by words and styles that we read in the past. You claim this isn't an algorithm, I don't think we really know how human intelligence works.
> You don't get words from what you read and study, you get ideas, and ideas can't be copyrighted.
That's completely oblivious to how humans actually read and study, of course you get words, if you are lucky you might also get ideas, but at least you get words (assuming ideas aren't words to begin with).
The government is eventually going to have to make a decision on this one way or the other, or we will just cede the whole of advancement to China (who won't bother with this debate and just do what works the best).
I disagree, but I doubt we're going to get anywhere by discussing it. It seems to me that you are either seriously undervaluing human intelligence, or seriously overvaluing LLMs.
As AI technology evolves, I would expect that the connections AIs have with the external world will become much more extensive, and that will make a huge difference as well. But current LLMs simply don't have that, and it shows.
But you don't do that by computing relative word frequencies in the content and then running an algorithm to generate further words with the same relative frequencies. When human efforts come too close to that it is called plagiarism. You don't get words from what you read and study, you get ideas, and ideas can't be copyrighted. You just have to express the ideas in your own words.
> what does it mean when a machine does that instead?
The machine is doing exactly the thing that, when human efforts come too close to doing it, is called plagiarism. The machine isn't operating with ideas; it doesn't even have a concept of "ideas". So it is not doing what you are doing--it is not taking ideas and expressing them in its own words. It's literally just shuffling the words.