Software that does this is running on millions of Android phones. After you do OK Google voice enrollment, they calculate a speaker representation embedding for you. When the voice activity detector detects voice, it passes the audio through VoiceFilter-Lite https://google.github.io/speaker-id/publications/VoiceFilter... along with your speaker embedding, and the model is trained to return only your speech. The original, more computationally expensive model is called VoiceFilter and I was so amazed when I saw the examples on this page: https://google.github.io/speaker-id/publications/VoiceFilter...