I think the Whisper models need to predict end-of-turn based on content. And if it still gets input after the EOT, it can just drop the LLM generation and start over at the next EOT.