News for hackers. Stuff that matters.
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models by Majromax on Monday April 10, 2023 @08:11PM (#35519035)
Highest posssible in which combination, though? If you’re fine tuning a model with N layers, then you could apply LoRA to any or all of them. Maybe it’s better to concentrate effort unevenly, in which case a uniform increase of adaptation rank (to compute budget) could still be subpar.
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models by whimsicalism on Monday April 10, 2023 @10:04PM (#35519820)
Right but the way that this paper proposes determining the best rank is by training a LoRA with the full rank.