In a job I had a couple of decades ago, I did work very much like this (not with music, but along the same lines). This certainly can be done, but your method isn't the way to do it.

There are two primary methods of accomplishing this. The fast and dirty method is to employ a notch filter to exclude all frequencies outside of those used by the human speaking voice (which, fortunately, is a rather narrow frequency range). The second is to perform some fancy mathemetics to be more selective about what you're filtering out and when. There are a number of different approaches to this, but they typically involve autocorrelations or crosscorrelations.

Both methods will cause a degradation of the audio you want to keep (the quick and dirty method will cause more degradation than the sophisticated methods). But the Q&D is often good enough, and is often used by karaoke machines and the like.

Simply trying to subtract out the known signal (as you were doing) is technically plausible -- and is done for certain types of signals -- but is very hard to do effectively. There are too many environmental effects that will distort the signal and make naive subtraction ineffective.