Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by vladf on Monday April 10, 2023 @05:05PM
(#35517406)
The optimal rank could differ across layers
[ Parent
| on HN ]
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by whimsicalism on Monday April 10, 2023 @05:26PM
(#35517576)
I would be shocked if the "optimal rank" in terms of performance wouldn't be using the maximum rank from the DynLoRA across all layers.
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by vladf on Monday April 10, 2023 @06:26PM
(#35518131)
Err, I suppose trivially, the higher rank terms include the lower-rank subnets, so they dominate in terms of quality.
But if you have some capacity constraint (e.g., memory, I guess?) then you can imagine dynamic rank allocation helping in the case where the maximum rank across all layers isn't within budget.
It's a bit of a stretch though, I agree
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by whimsicalism on Monday April 10, 2023 @07:46PM
(#35518842)
As someone else mentioned [0], the procedure would basically be to train a DyLoRA for an initial few iterations, then do a search among the layers to find the best scoring combination of ranks, and then train pruned to just use those ranks to completion.
Seems complicated but I could see it being useful potentially.