Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
by whimsicalism on Monday April 10, 2023 @05:38PM
(#35517712)
It's very different, but hard to distill in a comment. They use a new regularization technique to basically create a LoRA with dynamically adjustable rank.
It seems like they use a fixed-distribution controller for training. It’d be nice to see why it’s worth deviating from the original RL paradigm.