Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by Majromax on Monday April 10, 2023 @08:11PM
(#35519035)
Highest posssible in which combination, though? If you’re fine tuning a model with N layers, then you could apply LoRA to any or all of them. Maybe it’s better to concentrate effort unevenly, in which case a uniform increase of adaptation rank (to compute budget) could still be subpar.
[ Parent
| on HN ]
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by whimsicalism on Monday April 10, 2023 @10:04PM
(#35519820)
Right but the way that this paper proposes determining the best rank is by training a LoRA with the full rank.