News for hackers. Stuff that matters.
Re:OpenAI Audio Models by a-r-t on Thursday March 20, 2025 @03:07PM (#43427587)
Hi Jeff, are there any plans to support dual-channel audio recordings (e.g., Twilio phone call audio) for speech-to-text models? Currently, we have to either process each channel separately and lose conversational context, or merge channels and lose speaker identification.
Re:OpenAI Audio Models by ekzy on Thursday March 20, 2025 @05:30PM (#43429205)
I’m not entirely sure what you mean but twilio recordings supports dual channels already
Re:OpenAI Audio Models by a-r-t on Thursday March 20, 2025 @06:01PM (#43429479)
Transcribing Twilio's dual-channel recordings using OpenAI's speech-to-text while preserving channel identification.
Re:OpenAI Audio Models by ekzy on Thursday March 20, 2025 @06:08PM (#43429554)
Oh I see what you mean that would be a neat feature. Assuming you can get timestamps though it should be trivial to work around the issue?
Re:OpenAI Audio Models by a-r-t on Thursday March 20, 2025 @10:50PM (#43431365)
There are two options that I know of:

1. Merge both channels into one (this is what Whisper does with dual-channel recordings), then map transcription timestamps back to the original channels. This works only when speakers don't talk over each other, which is often not the case.

2. Transcribe each channel separately, then merge the transcripts. This preserves perfect channel identification but removes valuable conversational context (e.g., Speaker A asks a question, Speaker B answers incomprehensively) that helps model's accuracy.

So yes, there are two technically trivial solutions, but you either get somewhat inaccurate channel identification or degraded transcription quality. A better solution would be a model trained to accept an additional token indicating the channel ID, preserving it in the output while benefiting from the context of both channels.

Re:OpenAI Audio Models by claiir on Saturday March 22, 2025 @06:12AM (#43444690)
(2) is also significantly harder with these new models as they don’t support word timestamps like WHISPR.

see > Other parameters, such as timestamp_granularities, require verbose_json output and are therefore only available when using whisper-1.

Re:OpenAI Audio Models by jeffharris on Thursday March 20, 2025 @07:26PM (#43430209)
this has been coming up often recently. nothing to announce yet, but when enough developers ask for it, we'll build it into the model's training

diarization is also a feature we plan to add

Re:OpenAI Audio Models by a-r-t on Thursday March 20, 2025 @10:55PM (#43431389)
Glad to hear it's on your radar. I'd imagine phone call transcription is a significant use case.