News for hackers. Stuff that matters.
Re:LoRA Learns Less and Forgets Less by chaos_emergent on Friday May 17, 2024 @10:45AM (#40390514)
The findings are that the best fine-tune performance comes from fine-tuning all weights, followed my MLPs, followed by attention heads, using LoRA. Authors assert that the performance difference is based on the target module of the NN.

Isn’t an equally valid argument that MLPs tend to constitute a greater number of weights in transformer networks than attention heads, and the performance difference can be traced to a greater number of weights having freedom to change? I’d be curious to know if randomly choosing a subset of matrices to train, regardless of where they are in the network, would provide analogous performance to LoRA on a specific module with comparable learnable weights.

Re:LoRA Learns Less and Forgets Less by chaos_emergent on Friday May 17, 2024 @10:48AM (#40390541)
as a follow up curiosity, has anyone tried using LoRA on the entire model for pretraining to compare regular training model performance to LoRA?
Re:LoRA Learns Less and Forgets Less by buildbot on Friday May 17, 2024 @12:27PM (#40391571)
Yes, I’ve tested this out. It does train, but the scaling doesn’t seem to pan out. It’ll perform slightly better than the number of trainable parameters, but never improves as you scale, so for now there’s no benefit.
Re:LoRA Learns Less and Forgets Less by sp332 on Friday May 17, 2024 @12:10PM (#40391394)
Do you mean leaving most of the model in its initial, randomised state and only training a LoRA?
Re:LoRA Learns Less and Forgets Less by buildbot on Friday May 17, 2024 @12:29PM (#40391600)
I’ve tested specifically this (on my personal time) :) It will train but I found the loss is proportional to the number of trainable parameters. So roughly to hit the performance of a standard 70m param model, you need to train ~70m lora params anyway.
Re:LoRA Learns Less and Forgets Less by cheald on Friday May 17, 2024 @01:25PM (#40392242)
It's worse than that, because lora requires two matrices per layer. At full rank, you have an additional NxN parameters to learn versus full finetuning, where N is min(input_features, output_features).

For example, tuning a layer of 128 in x 256 out is 32k params. Learning a full-rank lora for that layer would be two matrices of 128x128 and 128x256 = 48k params.

Re:LoRA Learns Less and Forgets Less by buildbot on Friday May 17, 2024 @03:18PM (#40393316)
Yeah, exactly. Though the 48k param lora might be as good as a 48k param layer of higher rank, I haven't looked into that case really.
Re:LoRA Learns Less and Forgets Less by cabidaher on Friday May 17, 2024 @10:56AM (#40390640)
This paper [1] does atempt that and reports similar performance compared to conventional pre-training. However, they do start off by doing a normal full-rank training and claim that it is needed to 'warm start' the training process.

[1] https://arxiv.org/abs/2307.05695

Re:LoRA Learns Less and Forgets Less by danielhanchen on Friday May 17, 2024 @01:03PM (#40391974)
Oh yes this paper! The main issue is the scaling of the A and B LoRA matrices. Some papers show scaling the B matrix with larger learning rates (LoRA+) could be beneficial. DoRA for eg learns an auto scaling vector of numbers which tries to alleviate these issues.

Galore might be more equivalent to full pretraining with the gradients being low rank.

Re:LoRA Learns Less and Forgets Less by whimsicalism on Friday May 17, 2024 @02:23PM (#40392790)
i would be shocked if this worked well
Re:LoRA Learns Less and Forgets Less by danielhanchen on Friday May 17, 2024 @01:30PM (#40392300)
I think the QLoRA paper https://arxiv.org/pdf/2305.14314 paper also showed LoRA on all MLP + Attention layers > all MLP layers > just Attention layers.

Other papers show finetuning a select few layers can also work well.

Re:LoRA Learns Less and Forgets Less by 3abiton on Friday May 17, 2024 @08:01PM (#40395346)
Any real world performance comparison between QLoRa and LoRa?
Re:LoRA Learns Less and Forgets Less by danielhanchen on Saturday May 18, 2024 @03:10AM (#40397034)
The QLoRA paper itself provided some cool benchmarks across many many experiments - QLoRA is near equivalent to LoRA, with it sometimes exceeding or losing 1-2% accuracy (it depends on the use case)
Re:LoRA Learns Less and Forgets Less by thepasswordis on Friday May 17, 2024 @11:06AM (#40390728)
I really wish people would be more careful about choosing names for these things.

LoRa has been a popular wireless protocol for like 10 years.

Re:LoRA Learns Less and Forgets Less by sva_ on Friday May 17, 2024 @11:16AM (#40390829)
Yes, but this is LoRA, clearly not LoRa.
Re:LoRA Learns Less and Forgets Less by squarefoot on Friday May 17, 2024 @11:59AM (#40391278)
PP has a point though. I entered "LoRA" on Google, DuckDuckGo, Startpage and Bing, and all returned results in all first pages were about the communication protocol (1). They could have inferred my interests from previous searches, but I never used Bing in the last year or so, so it seems to me someone didn't care about name clashes.

(1) well, except Google which -surprise- returned about mid page an ad of a local quite expensive chandeliers brand called "LORA".

Re:LoRA Learns Less and Forgets Less by sva_ on Friday May 17, 2024 @04:19PM (#40393808)
I usually just add a term like 'ml' or 'nn' after my search to give the machine context and it is sufficient in most cases.
Re:LoRA Learns Less and Forgets Less by refulgentis on Saturday May 18, 2024 @12:02AM (#40396368)
Wait until you find out its a name too
Re:LoRA Learns Less and Forgets Less by johnisgood on Saturday May 18, 2024 @08:29AM (#40398452)
Most of the time you have to add a keyword as to what it is related. We cannot expect everything to have unique names, unless we are perfectly fine with random pronounceable strings as names.
Re:LoRA Learns Less and Forgets Less by atherton33 on Friday May 17, 2024 @02:09PM (#40392648)
I think you mean LoRa®, a registered trademark of Semtech® Corporation, for use only with permission and within specific guidelines. https://www.semtech.com/uploads/company/FAQ-for-Use-of-LoRa-...
Re:LoRA Learns Less and Forgets Less by mbirth on Friday May 17, 2024 @10:11PM (#40395996)
Not to be confused with Semtex who sell a completely different kind of problem solver.
Re:LoRA Learns Less and Forgets Less by noisy_boy on Saturday May 18, 2024 @12:53AM (#40396525)
Isn't Semtex an explosive aka RDX?
Re:LoRA Learns Less and Forgets Less by yau8edq12i on Saturday May 18, 2024 @01:35AM (#40396662)
Yes, you got the joke.
Re:LoRA Learns Less and Forgets Less by mobilemidget on Friday May 17, 2024 @02:11PM (#40392667)
I addressed the same in a previous 'lora' post on HN. For me the name is already reserved for the radio telecommunication meaning. Nothing going to change that.
Re:LoRA Learns Less and Forgets Less by renewiltord on Friday May 17, 2024 @02:29PM (#40392879)
Seriously, that was a terrible name for the wireless system since it's been used by the Loyola Online Records Access system for half a decade or more before the radio company shamelessly copied the name.
Re:LoRA Learns Less and Forgets Less by dheera on Friday May 17, 2024 @03:18PM (#40393315)
Not sure about "popular"

99% of ML engineers wouldn't know what it is.

Re:LoRA Learns Less and Forgets Less by enlyth on Friday May 17, 2024 @03:37PM (#40393478)
I'm not an ML engineer and the only reason I know that the wireless protocol exists is because in every HN article, there's a comment repeating the same complaint
Re:LoRA Learns Less and Forgets Less by Findecanor on Friday May 17, 2024 @03:53PM (#40393618)
99% of the engineers who are still working in ML (Machine Language) would.

A much smaller percent among those who write in ML (the functional programming language) probably, though.

Re:LoRA Learns Less and Forgets Less by nerdponx on Friday May 17, 2024 @04:14PM (#40393775)
[delayed]
Re:LoRA Learns Less and Forgets Less by goodpoint on Friday May 17, 2024 @05:13PM (#40394269)
...but they should know how to use search engines...
Re:LoRA Learns Less and Forgets Less by bryanrasmussen on Saturday May 18, 2024 @01:38AM (#40396680)
at some point we are going to run out of easily pronounceable abbreviations that are unique. Perhaps that point is actually in the past and we should just acknowledge it and move on. Although I guess it could have been Lorall - oops, that's a character in World of Warcraft.
Re:LoRA Learns Less and Forgets Less by dheera on Saturday May 18, 2024 @03:16AM (#40397053)
Old concepts become obsolete anyway. People can start reusing VCR, etc.
Re:LoRA Learns Less and Forgets Less by marcinzm on Saturday May 18, 2024 @03:17AM (#40397055)
As should have the IoT people to not conflict with the decades old LORA name used for Level of Repair Analysis.
Re:LoRA Learns Less and Forgets Less by barfbagginus on Monday May 20, 2024 @10:14PM (#40423314)
Anyone with some starch to their collar now knows that lora is both a wireless technology and an ANN fine tuning method.

It's virtually impossible to be sent for a loop if you're engaging critically. The two domains solve very different problems and have no overlapping concepts. The way we talk about each topic is very distinct.

Re:LoRA Learns Less and Forgets Less by bmitc on Saturday May 18, 2024 @01:44AM (#40396695)
Not much of a surprise there. The ML culture is to reinvent names for everything.
Re:LoRA Learns Less and Forgets Less by marcinzm on Saturday May 18, 2024 @03:15AM (#40397049)
And LORA stood for Level of Repair Analysis since the 70s.
Re:LoRA Learns Less and Forgets Less by onion2k on Saturday May 18, 2024 @07:44AM (#40398185)
The scale of human knowledge, and the pace we're increasing it at, means we probably can't have unique names for everything any more if we aim for short, pronounceable things. "Lora" must have dozens of meanings across different domains. The fact you recognize it from another one is just something you have to deal with.
Re:LoRA Learns Less and Forgets Less by BaculumMeumEst on Saturday May 18, 2024 @08:44AM (#40398546)
The other side of this is that if you become paralyzed by decisions because 0.001% of people are bothered by it, you're not gonna make it.
Re:LoRA Learns Less and Forgets Less by dsjoerg on Saturday May 18, 2024 @12:20PM (#40400133)
The overlap in the Venn Diagram of people who care about LoRA and people who care about LoRa is extremely small. Your problem is not typical of people in the Machine Learning field. That's why they didn't care, or more likely this issue didn't occur to the first 50 people who saw the name LoRA.
Re:LoRA Learns Less and Forgets Less by Turing_Machine on Saturday May 18, 2024 @12:44PM (#40400266)
Think of Apple naming its spreadsheet "Numbers" and its word processor "Pages" (or, for that matter, the name "Apple" itself. Or "Windows", "Word", "Access"...).

And yet (as others have noted) adding another word or two to give a bit of context is usually enough that web searches work.

Search engines are pretty clever nowadays, except when they've been deliberately dumbed-down (cough... Google...).

Re:LoRA Learns Less and Forgets Less by ssl-3 on Friday May 17, 2024 @12:20PM (#40391500)
What can we learn about Low Rank Acronyms today?
Re:LoRA Learns Less and Forgets Less by hybridtupel on Friday May 17, 2024 @10:21AM (#40390259)
This is about Low-rank adaptation. Not to be confused with LoRa the long range proprietary radio communication technique, which hopefully doesn't learn at all.
Re:LoRA Learns Less and Forgets Less by martinky24 on Friday May 17, 2024 @10:54AM (#40390604)
"Why the hell is LoRa learning" was indeed my first thought...
Re:LoRA Learns Less and Forgets Less by HeatrayEnjoyer on Saturday May 18, 2024 @02:08AM (#40396791)
This is how the subs were knocked offline in Terminator III
Re:LoRA Learns Less and Forgets Less by gregmac on Friday May 17, 2024 @10:22AM (#40390274)
This is "Low-Rank Adaptation", "a widely-used parameter-efficient finetuning method for large language models."

Not to be confused with LoRa ("long range") [1], an Internet of Things radio technology.

[1] https://en.wikipedia.org/wiki/LoRa

Re:LoRA Learns Less and Forgets Less by chaos_emergent on Friday May 17, 2024 @10:28AM (#40390329)
Isn’t this fairly obvious after a two second glance at the abstract
Re:LoRA Learns Less and Forgets Less by SubiculumCode on Friday May 17, 2024 @10:37AM (#40390436)
This is Low-rank adaptation. Not to be confused with Lake of the Ozarks Recreation Area.
Re:LoRA Learns Less and Forgets Less by 0cf8612b2e1e on Friday May 17, 2024 @01:36PM (#40392354)
Apparently constructed in 1929. You think those wireless people would have been more careful when they reappropriated the name.
Re:LoRA Learns Less and Forgets Less by chriskanan on Friday May 17, 2024 @01:54PM (#40392509)
This study is great and addresses a question I've had about LoRA for a while.

In a continual learning paper from last year, I found LoRA was extremely effective for faster fine-tuning and not forgetting the original dataset:

https://arxiv.org/abs/2306.01904

Re:LoRA Learns Less and Forgets Less by rzzzt on Friday May 17, 2024 @04:46PM (#40394058)
This paper has 12 authors, which fascinates me to no end for some unexplainable reason. How does it work? Is it a common occurrence to have this many people working on a submission? Did each of them get at least a paragraph in edgewise?
Re:LoRA Learns Less and Forgets Less by PeterisP on Friday May 17, 2024 @07:55PM (#40395299)
The general criteria for authorship require including the people who worked on the experiments and data for the paper, which can be more important contribution than most of the text in that paper. In other experimental fields, there are papers with dozens or even hundreds of authors, because it can take many people to get to a measurement of a single number in the paper.
Re:LoRA Learns Less and Forgets Less by rzzzt on Saturday May 18, 2024 @05:05AM (#40397531)
Thanks, this is the bit I've been missing.
Re:LoRA Learns Less and Forgets Less by repsak on Friday May 17, 2024 @04:57PM (#40394152)
I raise you the Gemini paper https://arxiv.org/abs/2312.11805
Re:LoRA Learns Less and Forgets Less by guyomes on Friday May 17, 2024 @05:17PM (#40394297)
Re:LoRA Learns Less and Forgets Less by jpgvm on Saturday May 18, 2024 @10:17AM (#40399271)
Goes to show how much money is being poured into this stuff.
Re:LoRA Learns Less and Forgets Less by SubiculumCode on Friday May 17, 2024 @06:24PM (#40394712)
For a serious answer, this is how it works in my field A researcher gets a grant with 3-7 co-investigators. This generates a bunch of data and other resources that will support 10 or more papers. Coinvestigators and PIs will ask their postdocs and grad students to write up a paper. PIs and co-Is go on every paper...because it's a paper from their grant. Then the 1 to 4 grad students and post-docs go on the paper, depending on their specific material contributions to the work, be it analysis, conception, or execution, or writing. The numbers can stack up.
Re:LoRA Learns Less and Forgets Less by yau8edq12i on Saturday May 18, 2024 @01:37AM (#40396673)
Wait until you learn that the paper on the LHC has more than 5000 authors: https://www.nature.com/articles/nature.2015.17567
Re:LoRA Learns Less and Forgets Less by iudexgundyr on Friday May 17, 2024 @04:53PM (#40394120)
I feel like this is a trivial conclusion. Keeping the rank low in the optimization is a common regularization technique.
Re:LoRA Learns Less and Forgets Less by Saris on Friday May 17, 2024 @09:31PM (#40395827)
What does LoRa have to do with LLMs? Whoever named this thing screwed up big time.
Re:LoRA Learns Less and Forgets Less by yinser on Saturday May 18, 2024 @03:12PM (#40401266)