News for hackers. Stuff that matters.
Re:Language Models Are Super Mario: Absorbing Abilities from... by og_kalu on Saturday April 06, 2024 @01:52PM (#39954103)
Image models merging is kind of meh though. You can't really merge two SD models trained on different styles with different keywords and get a model that knows both independently.
Re:Language Models Are Super Mario: Absorbing Abilities from... by smusamashah on Saturday April 06, 2024 @02:00PM (#39954178)
Never done that myself but always thought that was the point of these merges. May be it doesn't understand keywords after merge but I think they do keep the styles in some form. What's the point of merges if that wasn't possible. People sometimes share how much of a model they merged by a factor.
Re:Language Models Are Super Mario: Absorbing Abilities from... by og_kalu on Saturday April 06, 2024 @02:19PM (#39954311)
Let's say i trained a model on the artworks of fiona staples that is invoked by typing "fiona staples style". Then i trained a model on the artworks of james daly that is invoked by "james daly style".

What i want when i merge such models is a model that can generate in the artsyle of either fiona and james independently or a mix of both if i specify both keywords in the same prompt.

Currently, if you merge these 2 models and generate "busy city street, fiona staples style", you will not get a model that can generate works in fiona's style, you will just get a model that will generate an odd mix of fiona and daly even if you only specify one of them.

It means you either need to train a million different models for a million different concepts with no chance of cross usage (e.g x person wearing y clothes in z style will not be possible) or train those concepts at the same time, which becomes very cumbersome requiring a retrain of n+1 concepts on a fresh model anytime you want to introduce a new concept.

Oh, and training on x then training on y doesn't work in practice either because the model will mostly forget x learning y.

Re:Language Models Are Super Mario: Absorbing Abilities from... by zingelshuher on Saturday April 06, 2024 @10:29PM (#39957659)
Intuitively looks like models should be close enough, or sparse enough for merge to work. I wonder if MoE experts can be merged(?)