They tend to not have the right base of training materials - the models are constrained not just by the lobotomization but by some specialized writing not being part of the corpus. Training a lora on a specific corpus - fan fic, alt.sex.stories, or a persons specific writing, can help induce behavior or language choices specific to that corpus. You can also use context injection but I’ve found combining a model weight adjustment with priming the context to be the most effective.

I think over time it’ll become standard practice to train special purpose models on top of general purpose models. If I have a very specific task domain a model tuned to that domain will less often wander off and will be more likely to respond in the desired way.