By finetuning a model (correctly) you will retain its capability while shifting the distribution to what you desire, as I have said before "it is only because of these people that it is worth using models like SD".