This is the kind of thing where augmenting the NN with an actual raw audio buffer would allow it to mix in some of the signal from the past quite easily.