Re:Implementing Weight-Decomposed Low-Rank Adaptation (DoRA)...
by rasbt on Sunday February 18, 2024 @04:33PM
(#39423608)
Thanks for the feedback. Clarifying definitely wouldn't hurt. Added a paragraph and new figure at the top of the DoRA section: https://magazine.sebastianraschka.com/i/141797214/introducin...
I haven't tried what you were suggesting, but that sounds actually plausible. Interesting idea!
Not related to the article but tangentially relevant, would it be possible to train a LoRA or DoRA with a high rank, and then use SVD to see if the rank is too high and truncate to a better value of r? Maybe use different ranks for different layers after some training?