Re:LoRA: Low-Rank Adaptation of Large Language Models
by gliptic on Friday March 24, 2023 @12:51PM
(#35291890)
Correct me if I'm wrong, but I think you still need to compute gradients of non-trained weights in order to compute the gradients of the LoRA weights. What you don't have to do is store and update the optimizer state for all those non-trained weights.
[ Parent
| on HN ]
Re:LoRA: Low-Rank Adaptation of Large Language Models
by stu2b50 on Friday March 24, 2023 @12:56PM
(#35291965)
I mean the derivative of a constant is 0. So if all of the original weights are considered constants, then computing their gradients is trivial, since they’re just zero.
Re:LoRA: Low-Rank Adaptation of Large Language Models
by jprafael on Friday March 24, 2023 @01:17PM
(#35292306)
Computing gradients is easy/cheap. What this technique solves is that you no longer need to store the computed values of the gradient until the backpropagation phase, which saves on expensive GPU RAM, allowing you to use commodity hardware.