News for hackers. Stuff that matters.
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models by whimsicalism on Monday April 10, 2023 @05:28PM (#35517617)
I can see that. But are we sure that a rank-based difference that doesn't manifest early in the training process won't manifest as you get further along? See also 'grokking' [0]

[0]: https://arxiv.org/abs/2201.02177

Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models by fancyfredbot on Monday April 10, 2023 @05:50PM (#35517823)
Not sure there's any way to know beforehand whether that would happen but the advantage of DyLoRA is that at least you will know afterwards whether you really needed the full rank whereas with LoRA you wouldn't? In some cases that might not be valuable information but I guess you'd rather know than not.