This is changing very rapidly. You don’t need that anymore
https://twitter.com/karpathy/status/1661417003951718430?s=46
There’s an inverse Moore’s law going on with compute power requirements for AI models
The required compute power is decreasing exponentially
Soon (months, maybe a year), people will be training models on their gamer-level GPUs at home, maybe even on their computer CPUs
Plus all the open and publicly available models both on HuggingFace and on GitHub
For training LAMA itself, Meta I believe said it cost them $5 million. That is actually not that much, but I believe that is just the cost of running the cluster for the the duration of the training run. I.e, doesn't include cost of cluster itself, salaries, data, etc.
Almost by definition, the research frontier work will always require big clusters. Even if in a few years you can train a GPT4 analogue in your basement, by that time OpenAI will be using their latest cluster to train 100 trillion model parameters.
The point is that this is unstoppable
Many ways to regulate that. For instance, require tracking of GPUs and that they must connect to centralized servers for certain workloads. Or just go ahead and nationalize and shutdown NVDA.
(And no, fine-tuning LAMA based models is not state of the art, and is not where the real progress is going to come from)
And even if all the regulation does is slow down progress, every extra year we get before recursively self improving AGI increases the chances of some critical advance in alignment and improves our chances a little bit.