Well, there is CrowdGPT[0] which is a fully open and distributed attempt to train a small 1B parameter LLM from scratch with open data. They've started just a few days ago but they're already at 13.0B / 100.0B tokens for the pretraining phase.

Of course training something like a 1T LLM will probably take some time :-P but with enough people contributing it might be possible to train something small - and if the project reaches consensus, it might be adopted by some institutions with a bit more available compute power too (e.g. universities).

If i had a second PC (i don't really trust the project right now to put their auto-updating client on my main PC - maybe if it grows and there are more people behind it or there is some better governance than "trust me bro" i'd put it in a container or something) with some GPU power i might have contributed a bit too since i like the idea.

[0] https://www.crowdgpt.net/#status

That's not happening without significant funding. The cost of the primary training run for frontier models is in the hundreds of millions of dollars, and is increasing with every iteration. The Astra pre-train likely cost close to a billion dollars.

The Chinese open-source models (which are 1-2 generations behind the frontier) cost in the low tens of millions to train. This is largely because of a lack of compute. They achieve comparable performance to frontier models via distillation.