News for hackers. Stuff that matters.
Re:DyLoRA: Parameter Efficient Tuning of Pre-Trained Models by whimsicalism on Monday April 10, 2023 @07:46PM (#35518842)
As someone else mentioned [0], the procedure would basically be to train a DyLoRA for an initial few iterations, then do a search among the layers to find the best scoring combination of ranks, and then train pruned to just use those ranks to completion.

Seems complicated but I could see it being useful potentially.

[0]: https://news.ycombinator.com/item?id=35517353

There are no comments on this comment yet.