It’s my understanding that, amazingly enough, blending the models is done by literally performing a trivial linear blend of the raw numbers in the model files.
Someone even figured out they could get great compression of specialized model files by first subtracting the base model from the specialized model (using plain arithmetic) before zipping it. Of course, you need the same base file handy when you go to reverse the process.
It is not typically possible to blend models like that, since the training process is (lateral) order insensitive, as far as the model goes.
Re:Stable Diffusion 2.0 Release
by ShamelessC on Thursday November 24, 2022 @04:37AM
(#33729385)
Is that still the case when all models have a common ancestor (i.e. finetuned) and haven’t yet overfit on new data?
I thought so too until found that there are quite a bit of literatures nowadays about "merging" weights, for example, this one: https://arxiv.org/pdf/1811.10515.pdf and also the OpenCLIP paper.
Edit: Looks like AUTOMATIC1111 can merge three checkpoints. I still don't know how it works technically, but I guess that's how it's done?
https://github.com/AUTOMATIC1111/stable-diffusion-webui