Re:The Coming of Local LLMs
by whimsicalism on Tuesday April 11, 2023 @04:52PM
(#35531380)
Even with LoRA the economics are not in your favor to buy a 4090 instead of cloud training.
Re:The Coming of Local LLMs
by thewataccount on Tuesday April 11, 2023 @04:57PM
(#35531446)
Actually yeah you're definitely right.
My perspective was for people who have other uses for them e.g. gaming or local inference. From a pure finance standpoint you're definitely right - you should rent and not buy a dedicated card. I think you'd need a few thousand hours to break even which is a few months 24/7.
So finetuning with LoRAs and a few other methods is fine on higher end consumer hardware like a 4090 and finishes in a reasonable amount of time - IMO definitely worth it if you're experimenting with this especially for the inference.
The base training though yeah I totally agree with you - train in the cloud, don't buy hardware when you need a month of 8x A100's or whatnot.