> Clearly a company with $5-5MM in the bank can’t train a competitive LLM from scratch...

Wait, are we sure?

I'm going to make the massive mistake of assuming we're compute bound instead of memory bound, and assume we can train at FP16 (which is a bad assumption because, of course, you're doing calculus where the little pieces you're adding up could get rounded to zero at FP16 pretty easily... although mixed precision FP32/FP16 training is possible).

Consumer GPUs like GeForce RTX 4090 can do 3e14 flop/s under certain conditions with fp16. They retailed for about $1600. It took reportedly 3e23 flop to train GPT-3. A year is 3e7 seconds. So the upfront cost of retail GPUs doing 3e23 fp16 operations in a single year is potentially as low as ~$50k (and about $20k worth of electricity). (FP32 peak is about a factor of 4 worse, so ~$200k.)

So it's not actually impossible to imagine a particularly clever approach to training that could maybe achieve competitive LLM training for less than $5 million in hardware costs. (except for the fact that compute isn't really the bottleneck, memory really is.)

It would be pretty funny if a start up took its first round and put a huge chunk of it into a crypto mining style ad hoc data center.

Exactly. I think we haven’t optimized for hardware cost much with LLM training. Crypto is kind of like that but taken to an extreme. There are some reasons why it’s not comparable (crypto mining is embarrassingly parallel, doesn’t need a lot of memory, tends to burn at least as much cost in electricity as on hardware, whereas LLM training is memory-bound, etc), but it shows kind of roughly what scrappy, hardcore computer hardware cost-optimization looks like at scale.

The main "issue" with this approach, is you can't iterate quickly. If your idea didn't pan out, you may have wasted some serious time on that one attempt. Not trying to knock it, just wanted to point out that there are potential downsides.

You don't have to restart training though. You can tune hyperparameters and fork one of the previous checkpoints.