Re:Replit - How to train your own LLM Models
by kkielhofner on Wednesday April 19, 2023 @10:40PM
(#35635718)
Ouch.
Ok, I'll try again because I read the article and watched the demo video from Mosaic:
- MosaicML claims to have some magic that efficiently autoscales with awareness of specific instance per hour pricing.
- MosaicML claims to have auto-optimize magic that makes training 3-7x faster.
- The incentives seem to be aligned because their value prop is likely "we'll markup 10% but save you 50%" (or whatever).
It's too bad Replit didn't provide the costs for this post in the same way Stanford, etc have done with Alpaca, Vicuna, etc. They probably didn't for from-scratch training because the answer to your question is almost certainly "it depends" or "if you have to ask you can't afford it" (for now).
Re:Replit - How to train your own LLM Models
by ftxbro on Wednesday April 19, 2023 @10:52PM
(#35635806)
Thanks I understand better now. Sorry my other comment was rude. Probably I'm just frustrated that I will never be able to make a sentient bot because it will cost literally a billion dollars, and I just have to sit on sidelines and watch the billionaires play with their sentient bots.
Thank you for your response! I appreciate your insights on fine-tuning and using open LLMs. However, I was hoping for more information on the cost of training LLMs from scratch. Could you please elaborate on that? Thank you again for your help!
original unfriendly comment below:
--
OK thank you but that doesn't help answer my questions at all.
> Looking at what they're doing here probably not as much as you think.
thanks I guess
> As you note [...] you can fine tune
yes I did note that, to preempt answers like 'you can just fine-tune instead of training from scratch bro' but i guess it didn't work
> I doubt anyone reading a blog post is attempting to train a "true" multi-billion param LLM from scratch.
thanks again