Wait, are we sure?

I'm going to make the massive mistake of assuming we're compute bound instead of memory bound, and assume we can train at FP16 (which is a bad assumption because, of course, you're doing calculus where the little pieces you're adding up could get rounded to zero at FP16 pretty easily... although mixed precision FP32/FP16 training is possible).

Consumer GPUs like GeForce RTX 4090 can do 3e14 flop/s under certain conditions with fp16. They retailed for about $1600. It took reportedly 3e23 flop to train GPT-3. A year is 3e7 seconds. So the upfront cost of retail GPUs doing 3e23 fp16 operations in a single year is potentially as low as ~$50k (and about $20k worth of electricity). (FP32 peak is about a factor of 4 worse, so ~$200k.)

So it's not actually impossible to imagine a particularly clever approach to training that could maybe achieve competitive LLM training for less than $5 million in hardware costs. (except for the fact that compute isn't really the bottleneck, memory really is.)