News for hackers. Stuff that matters.

A comment on Show HN: LunaLora: Multi-LoRA System to Combat Catastrophic Forgetting. Parent comment

Re:Show HN: LunaLora: Multi-LoRA System to Combat Catastroph... by SphericalCowww on Saturday April 04, 2026 @08:08AM (#47638335)
Just to understand what you mean, you are worried that there are 2 or more LoRAs that have almost identical content but are distinct only in timestamps. Yeah, that could be a problem...

Hmm..., perhaps the similarity for recency does need to be incorporated through orthogonality, but can somehow be implemented through weight preference, such that the new weight is influenced by the weight history.

Thanks, that is a very good point, will take that into consideration!

[ Parent | on HN ]
Re:Show HN: LunaLora: Multi-LoRA System to Combat Catastroph... by sensarts on Tuesday April 07, 2026 @09:33AM (#47675145)
Exactly, that's the worry. If two similar updates happen close in time, orthogonality forces them apart for no reason. I like your weight preference idea. Maybe just apply orthogonality when updates are actually different, not when they are nearly the same. Probably you should try that if not tried it already. :)
[ Parent | on HN ]