LoRA: Low-Rank Adaptation of Large Language Models

Re:LoRA: Low-Rank Adaptation of Large Language Models by eternalban on Friday March 24, 2023 @06:16PM (#35296436)

Great, we can get authoritative answers. (I'm trying to understand the ML space and have mostly done readings, not an expert.)

I am assuming you can have n LoRA fine-tunings, say each specializing in one aspect of a coherent task, with n summers, running in parallel, and then combine them at the end? Or more generally, does LoRA enable a sort of modularizing around a core (un-merged) model?

And curious if you ever tried merging 2 or more fine-tunings and then testing the resultant single model (merge all) against the original tests to check retention?

[ Parent | on HN ]

Re:LoRA: Low-Rank Adaptation of Large Language Models by edwardjhu on Friday March 24, 2023 @07:08PM (#35296885)

This paper tries something like that

https://arxiv.org/pdf/2202.13914.pdf

The gain isn't that significant. We don't understand what these low-rank updates represent, and they might not correspond to "skills" that humans have.

[ Parent | on HN ]