News for hackers. Stuff that matters.

A comment on Fine tune LLAMA3 on million scale dataset in consumer GPU using QLora, DeepSpeed. Parent comment

Re:Fine tune LLAMA3 on million scale dataset in consumer GPU... by anonymousDan on Thursday April 25, 2024 @06:05AM (#40155528)
Can you point to any literature on this by any chance? I would be really interested to see some in depth analysis.
[ Parent | on HN ]
Re:Fine tune LLAMA3 on million scale dataset in consumer GPU... by bradfox2 on Saturday May 04, 2024 @11:27PM (#40262053)
I don't have the arxiv link bookmarked, but there was a paper written on pretraining with Lora. It involved merging adapters back every n steps with good results.
[ Parent | on HN ]