I would love to help. Would even code up a prototype if you wanted :)
I like the idea of an RL predictor for interruption timing, although I think it might struggle with factual-correction interruptions. It could be a good way to make a very fast system, and if latency on the rest of the system is low enough you could probably start slipping in your "Of course", "Yeah, I agree", and "It was in March, but yeah" for truly natural speech. If latency is low you could just use the RL system to find opportunities to interrupt, give them to the LLM/LMM, and it decides how to interrupt, all the way from "mhm", to "Yep, sounds good to me", to "Not quite, it was the 3rd entry, but yeah otherwise it makes sense", to "Actually can I quickly jump on that? I just wanted to quickly [make a point]/[ask a question] about [some thing that requires exploration before the conversation continues]".
Tuning a system like this would be the most annoying activity in human history, but something like this has to be achieved for truly natural conversation so we gotta do it lol.
Maybe you can rely on some sort of speaker identification to sort this out?
Echo cancellation is non-trivial for sure.
I'd rather use speaker diarization and/or echo cancellation to solve the problem without needing the user to press any buttons.
That's how I've been thinking of doing it - seemed like you could use a much smaller GPT-J-ish model for that, and measure the relative probability of 'yes' vs 'no' tokens in response to a question like 'is the user done talking'. Seemed like even that would be orders of magnitude better than just waiting for silence.
The other way around is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is ready it starts over. If it finishes its response before any more words arrive (~1 second usually) it starts speaking. This is not ideal because the user might not be done speaking, of course. If the user continues speaking the bot will stop. But deciding when the user is done speaking (or if the bot should interrupt before the user is done) is a hard problem. It could possibly be done zero-shot using prompting of a LLM but you'd probably need a GPT-4 level LLM to do a good job and GPT-4 is too slow for instant response right now. A better idea would be to train a turn-taking model that predicts who should speak next in conversations. I don't know where to source a dataset for a model like that though.
Ultimately the end state of this type of system is a complete end-to-end audio-to-audio language model. There should be only one model, it should take audio directly as input and produce audio directly as output. I believe that having TTS and voice recognition and language modeling all as separate systems will not get us to 100% natural human conversation. I think that such a system would be within reach of today's hardware too, all that's really missing is the training dataset/procedure and some architecture bits to make it efficient.