This might not work at all but I wonder how it would work if you supplied it with a "mixed" language model. That is, a language model based around a person using multiple languages interchangably on the fly, simultaneously, instead of doing this modal switching stuff that causes so many headaches.