News for hackers. Stuff that matters.

A comment on ChatGPT can now see, hear, and speak – openai.com. Parent comment

Re:ChatGPT can now see, hear, and speak – openai.com by hk__2 on Monday September 25, 2023 @05:54PM (#37651250)
Don’t they do it already? There are a lot of languages where intonation is absolutely necessary to distinguish between some words, so I would be surprised that this not already taken into account by the major voice assistants.
[ Parent | on HN ]
Re:ChatGPT can now see, hear, and speak – openai.com by bobsmooth on Monday September 25, 2023 @07:25PM (#37652161)
In English, intonation changes the meaning of the word but not the word itself. From what I understand, in tonal languages tone changes the whole word. I don't think ML understands that difference yet.
[ Parent | on HN ]
Re:ChatGPT can now see, hear, and speak – openai.com by fragmede on Monday September 25, 2023 @09:35PM (#37653220)
Yeah they do. I was able to get ChatGPT-4 to transcribe 我哥哥高過他的哥哥, which says that they can. I did have to set the app to Chinese, and the original didn't work so I had to modify what I said slightly.

https://www.tiktok.com/t/ZT86psPxY/

Roughly translated, my older brother is taller than that other guy's older brother.

[ Parent | on HN ]
Re:ChatGPT can now see, hear, and speak – openai.com by modeless on Monday September 25, 2023 @10:18PM (#37653497)
Of course speech recognition works for Chinese. What it doesn't do is transcribe intonation and prosody in non-tonal languages. It's not even clear how one would transcribe such a thing as I'm not aware of a standard notation.
[ Parent | on HN ]
Re:ChatGPT can now see, hear, and speak – openai.com by fragmede on Friday September 29, 2023 @04:21AM (#37700786)
IPA format should cover that, no?
[ Parent | on HN ]
Re:ChatGPT can now see, hear, and speak – openai.com by modeless on Saturday September 30, 2023 @01:12AM (#37712928)
Maybe? I thought IPA was just phonetic but I see that it does have some optional prosody stuff that could in theory cover some of it. I'm not sure how standard or complete it really is in practice.

I haven't heard of any large datasets of IPA transcripts of speech with the detail necessary to train a fully realistic STT->LLM->TTS system. If you know of some that would be interesting to look at.

[ Parent | on HN ]