Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by huevosabio on Monday April 10, 2023 @04:35PM
(#35517083)
Yea, this is interesting but I can't see the immidiate value (not that there isn't).
Maybe if the "optimal rank" of LORA applies to any adaptation and you interested in training multiple adaptations for different use cases?
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by vladf on Monday April 10, 2023 @05:05PM
(#35517406)
The optimal rank could differ across layers
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by whimsicalism on Monday April 10, 2023 @05:26PM
(#35517576)
I would be shocked if the "optimal rank" in terms of performance wouldn't be using the maximum rank from the DynLoRA across all layers.
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by vladf on Monday April 10, 2023 @06:26PM
(#35518131)
Err, I suppose trivially, the higher rank terms include the lower-rank subnets, so they dominate in terms of quality.
But if you have some capacity constraint (e.g., memory, I guess?) then you can imagine dynamic rank allocation helping in the case where the maximum rank across all layers isn't within budget.
It's a bit of a stretch though, I agree
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by whimsicalism on Monday April 10, 2023 @07:46PM
(#35518842)
As someone else mentioned [0], the procedure would basically be to train a DyLoRA for an initial few iterations, then do a search among the layers to find the best scoring combination of ranks, and then train pruned to just use those ranks to completion.
Seems complicated but I could see it being useful potentially.
But nonetheless, training time improvements look interesting.
e: Oh I see, the training time improvement is compared to a grid search over the LoRA rank. Not for a single run.
I am not convinced that you shouldn't just train on the highest possible rank that you can with your compute budget. If you can train a DynLoRA with rank 8, why not just train a LoRA with that rank?