LoRA Learns Less and Forgets Less

Re:LoRA Learns Less and Forgets Less by danielhanchen on Friday May 17, 2024 @01:30PM (#40392300)

I think the QLoRA paper https://arxiv.org/pdf/2305.14314 paper also showed LoRA on all MLP + Attention layers > all MLP layers > just Attention layers.

Other papers show finetuning a select few layers can also work well.

[ Parent | on HN ]

Re:LoRA Learns Less and Forgets Less by 3abiton on Friday May 17, 2024 @08:01PM (#40395346)

Any real world performance comparison between QLoRa and LoRa?

[ Parent | on HN ]

Re:LoRA Learns Less and Forgets Less by danielhanchen on Saturday May 18, 2024 @03:10AM (#40397034)

The QLoRA paper itself provided some cool benchmarks across many many experiments - QLoRA is near equivalent to LoRA, with it sometimes exceeding or losing 1-2% accuracy (it depends on the use case)

[ Parent | on HN ]