News for hackers. Stuff that matters.
Re:The Coming of Local LLMs by thewataccount on Tuesday April 11, 2023 @04:40PM (#35531233)
This isn't necessarily true with LoRAs - a 4090 can train/compute the alpaca dataset with LoRA in under 6 hours (it might be 3, I forget what it was).

So finetuning with LoRAs and a few other methods is fine on higher end consumer hardware like a 4090 and finishes in a reasonable amount of time - IMO definitely worth it if you're experimenting with this especially for the inference.

The base training though yeah I totally agree with you - train in the cloud, don't buy hardware when you need a month of 8x A100's or whatnot.

Re:The Coming of Local LLMs by whimsicalism on Tuesday April 11, 2023 @04:52PM (#35531380)
Even with LoRA the economics are not in your favor to buy a 4090 instead of cloud training.
Re:The Coming of Local LLMs by thewataccount on Tuesday April 11, 2023 @04:57PM (#35531446)
Actually yeah you're definitely right.

My perspective was for people who have other uses for them e.g. gaming or local inference. From a pure finance standpoint you're definitely right - you should rent and not buy a dedicated card. I think you'd need a few thousand hours to break even which is a few months 24/7.