Roll to disbelief. That tweet is precisely about what I mentioned in my previous post that doesn't count: finetuning LAMA derived models. You are not going to contribute to the cutting edge of ML research doing something like that.
For training LAMA itself, Meta I believe said it cost them $5 million. That is actually not that much, but I believe that is just the cost of running the cluster for the the duration of the training run. I.e, doesn't include cost of cluster itself, salaries, data, etc.
Almost by definition, the research frontier work will always require big clusters. Even if in a few years you can train a GPT4 analogue in your basement, by that time OpenAI will be using their latest cluster to train 100 trillion model parameters.
It doesn’t matter
The point is that this is unstoppable
They aren’t saying that AI needs to stop existing, just that more effort needs to be put towards regulating it, monitoring it, aligning it, etc.
And what I’m saying is that it doesn’t matter and it is unstoppable
This is changing very rapidly. You don’t need that anymore
https://twitter.com/karpathy/status/1661417003951718430?s=46
There’s an inverse Moore’s law going on with compute power requirements for AI models
The required compute power is decreasing exponentially
Soon (months, maybe a year), people will be training models on their gamer-level GPUs at home, maybe even on their computer CPUs
Plus all the open and publicly available models both on HuggingFace and on GitHub