There are two primary methods of accomplishing this. The fast and dirty method is to employ a notch filter to exclude all frequencies outside of those used by the human speaking voice (which, fortunately, is a rather narrow frequency range). The second is to perform some fancy mathemetics to be more selective about what you're filtering out and when. There are a number of different approaches to this, but they typically involve autocorrelations or crosscorrelations.
Both methods will cause a degradation of the audio you want to keep (the quick and dirty method will cause more degradation than the sophisticated methods). But the Q&D is often good enough, and is often used by karaoke machines and the like.
Simply trying to subtract out the known signal (as you were doing) is technically plausible -- and is done for certain types of signals -- but is very hard to do effectively. There are too many environmental effects that will distort the signal and make naive subtraction ineffective.
My feeling is that it wouldn't work very well at all, because of things like the Doppler effect shifting the sound as you move the mic closer to or further from the source, and also the sampling on the microphone might be out of phase with the original recording. But the more I think about it, the more I think those might only be minor problems and it would actually work well?
EDIT: And the frequency response of the speaker & microphone are almost certainly non-uniform.
EDIT2: From a quick test, it doesn't even slightly work. Here's what I did:
1. Find a song on youtube
2. Save the song to an MP3 with https://www.bestmp3converter.com (or use youtube-dl - bestmp3converter.com probably uses youtube-dl)
3. Play the song on YouTube and record it using my phone
4. Load the MP3 and the recording from my phone in Audacity as 2 separate tracks
5. Amplify the phone recording until it looks like it matches the MP3 (Effect -> Amplify)
6. Delete some samples at the start of the phone recording so that the 2 tracks visually line up
7. Invert one of the tracks (Effect -> Invert)
8. Add the 2 tracks together (select both tracks, then Tracks -> Mix -> Mix and Render to New Track)
9. Mute the 2 original tracks and listen to the mixed result - it still sounds exactly like the song. It's not even slightly muffled. It sounds the same even if you don't invert one of the tracks. I think this is just much too far away from lining up sample-for-sample to be close to working.