It doesn't seem your oscillators are behaving correctly when you vary their frequency input. You seem to be doing something like phase = freq * time, which will break horribly when freq changes. Instead, you need to accumulate the phase over time based on whatever freq happens to be, like phase += freq * timeOfSingleSample.