News for hackers. Stuff that matters.
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models by vladf on Monday April 10, 2023 @05:02PM (#35517383)
How does this technique differ from the supernet optimization for one-shot NAS? https://proceedings.mlr.press/v80/bender18a.html

It seems like they use a fixed-distribution controller for training. It’d be nice to see why it’s worth deviating from the original RL paradigm.

Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models by whimsicalism on Monday April 10, 2023 @05:38PM (#35517712)
It's very different, but hard to distill in a comment. They use a new regularization technique to basically create a LoRA with dynamically adjustable rank.