As you note, with the plethora of open/open-ish LLMs today and LoRA + PEFT you can fine tune with low VRAM and pretty quickly so even a single A100 or whatever cloud GPUs are just fine. I've even seen people pull it off in reasonable time on super cheap T4s, A10s, etc.
I doubt anyone reading a blog post is attempting to train a "true" multi-billion param LLM from scratch.
Thank you for your response! I appreciate your insights on fine-tuning and using open LLMs. However, I was hoping for more information on the cost of training LLMs from scratch. Could you please elaborate on that? Thank you again for your help!
original unfriendly comment below:
--
OK thank you but that doesn't help answer my questions at all.
> Looking at what they're doing here probably not as much as you think.
thanks I guess
> As you note [...] you can fine tune
yes I did note that, to preempt answers like 'you can just fine-tune instead of training from scratch bro' but i guess it didn't work
> I doubt anyone reading a blog post is attempting to train a "true" multi-billion param LLM from scratch.
thanks again
Ok, I'll try again because I read the article and watched the demo video from Mosaic:
- MosaicML claims to have some magic that efficiently autoscales with awareness of specific instance per hour pricing.
- MosaicML claims to have auto-optimize magic that makes training 3-7x faster.
- The incentives seem to be aligned because their value prop is likely "we'll markup 10% but save you 50%" (or whatever).
It's too bad Replit didn't provide the costs for this post in the same way Stanford, etc have done with Alpaca, Vicuna, etc. They probably didn't for from-scratch training because the answer to your question is almost certainly "it depends" or "if you have to ask you can't afford it" (for now).
Training these models from scratch on your domain specific data is not as expensive as one might think. We have provided some cost estimates in our blogs.
https://www.mosaicml.com/blog/mosaicbert
https://www.mosaicml.com/blog/training-stable-diffusion-from...
[1]: https://twitter.com/jefrankle/status/1649060478910357504?s=4...