Big corp will always have more data. Even just the web, search engine have a big head start. Then there are "private" data that big provider use for "improving our product": emails, word documents, chat messages, non public social media posts,...

Yeah and the model will be useless when the courts determine it's illegal to use it commercially without having properly licensed the data.

Remember... Bits have color and AI models are a rainbow.

That’s extremely unlikely and impractical anyway. I’ve heard multiple times this mythical “courts” are going to change everything, what makes people think this?

“The courts” don’t have rules which apply globally for this sort of thing. My understanding is that even if the models were made illegal to host in America, there’s no shortage of other countries much looser copyright laws where one could hypothetically host such a thing.

There are clearnet sites which host entire movies and games and there have been since the dawn of the internet, wouldn’t you expect those would be bigger targets to go after if the powers were there? All that to say, I’m not bullish on the future global leverage of American copyright law, especially when it comes to AI models.

But courts have jurisdiction over their consumer markets, if a court in the US says it's illegal to use non-licenced data for training models and any product doing so can't be used/sold in the US you have lost access to a massive consumer market.

Case in point: GDPR, you can vacuum as much private data as you wish from non-EU citizens, the moment you touch EU-citizen data you must abide by the GDPR or you don't have access to a market of 400+ million of some of the wealthiest countries in the world.

You don't need global jurisdiction when your economical power is large enough.

Which leaves the citizens in countries without large enough economical power at the mercy of those corporations. Nothing new, you will say.

That's a useful number. Just yesterday we had an announcement of an open source system you can train with one GPU on a desktop for simple models. That could usefully use 16 or 24 GPUs, but it's not like you need a whole data center. It doesn't look like this is an industry with excessive monopoly potential.

The real issue for machine learning is that it lets you build systems that optimize for some metric. That's what corporations do. I've been pointing out lately that the ethical problems for AI in the near term look very much like those for corporations.

I’ve been thinking along the same lines as what you seem to have done. I looked into the CommonCrawl data and also the Pile and I’m not sure they are as large as people think. We also manage some of our own hardware including GPUs. It looked to me that with maybe .5 PB decent storage and 8 high end gpus it could be possible to train a decent LLM in a few months, how does that estimate compare to your experience?