Re:I made a talking emoji using regular emojis and JavaScript
by _nalply on Tuesday December 26, 2017 @03:32AM
(#16007744)
I'm Deaf and read lips.
Thanks for this extremely intriguing idea to use emoji to depict a talking person visually.
You know, some animated films get it right. Sometimes I clearly see utterances like «Thank you!», «Okay», but more often not. Especially older animations don't care and just let the character move the mouth in very simplistic ways: «Babbabbabaa Baababba ba baaa.»
Similarly for that emoji. It is moving the mouth in quite arbitrary ways. It looks like: «Sobbabbabee be <grin> seebee <grin> <frown> babbaa».
To solve this problem we need about twelve emoji for utterances: «ah», «ay», «ee», «oh», «oo», «b», «d», «f», «l», «n», «r» and «s». The «r» emoji must be animated (so we see the trilling tongue). The other sounds are either invisible like «k» or cover several different sounds like «b» also covering «m» and «p».
«Okay» would be rendered by: «oh», «oo», «oo» (standing for «k» wich is invisible) «ay» and «ee».
Edit: Add «th». This is an extremely simple sound to lipread. I remember my delight when I learnt English in my youth and in an movie suddenly realised that an actor said «Thank you».
Re:I made a talking emoji using regular emojis and JavaScript
by twiss on Tuesday December 26, 2017 @03:52AM
(#16007791)
Halfway through the article, he shows a picture with mouths for a-z, sh and th: https://cdn-images-1.medium.com/max/1600/0*gwbWfYm-iQIPEXXP%... There are others if you search images for "lip sync animation". It would be interesting to see if the result is understandable if you simply replace the emojis with those images.
Re:I made a talking emoji using regular emojis and JavaScript
by nurettin on Tuesday December 26, 2017 @09:13AM
(#16008661)
That image does not map to letters, it maps to the phonetic alphabet. English has different phonetics for words when compared to their written counterparts. For example "money" needs to be converted to "mahnee" before being converted to emoji counterpart. Otherwise it is gibberish.
Re:I made a talking emoji using regular emojis and JavaScript
by yesenadam on Tuesday December 26, 2017 @10:28AM
(#16008962)
'mah-' is US English, not English :P It's 'mah' in one country.
p.s. Seems every day on here I'm reminding people from the US that their country is not the world, their version of English is spoken and spelt in just one country, etc etc etc. Maybe I should give up doing that.
The most classic maybe was a linguist (yes, amazing) who on her blog was gloating that her variety of US English was particularly rich, in having almost every vowel sound in English. I looked at her list.. it didn't include the 'o' sound in 'pot'..
Re:I made a talking emoji using regular emojis and JavaScript
by viridian on Tuesday December 26, 2017 @10:54AM
(#16009097)
Either you don't understand that he means the pheonetic alphabet, or you say money in a way that is foreign to almost any person in the UK as well as the US. Do you say mooney or mohney? If not I think your phoneme almost certainly must be mah.
Re:I made a talking emoji using regular emojis and JavaScript
by yesenadam on Friday December 29, 2017 @07:35PM
(#16033024)
Hi, sorry about the delay. I don't know what you mean, 'he means the phonetic alphabet'. I say the first syllable to rhyme with 'bun'. That's surely fairly standard (non-US) English. It's mysterious to me that that's not one of your alternatives.
Re:I made a talking emoji using regular emojis and JavaScript
by viridian on Tuesday December 26, 2017 @12:09PM
(#16009465)
The former is the most common way for Americans to pronounce it, and matches the mouth shape of ah, and the latter, while not common in the US, also matches the ah mouth shape. I really don't know how yesenadam is pronouncing it such that his mouth doesn't take the ah shape.
Re:I made a talking emoji using regular emojis and JavaScript
by _nalply on Tuesday December 26, 2017 @04:50AM
(#16007926)
Lipreading is different. For example the «sh» sound is made by slightly pursing the lips but not all speakers do this. In German it gives a very clear visual picture.
Re:I made a talking emoji using regular emojis and JavaScript
by falsedan on Tuesday December 26, 2017 @07:01AM
(#16008232)
I would only purse my lips if I were performing a shush. In words like ‘hash’ I’d just close my mouth slightly & leave lips parted and wide from the ‘ha’ sound.
I agree that the mouth shapes are garbage in this demo. A shame because that is the most interesting & important part of animating speech.
Re:I made a talking emoji using regular emojis and JavaScript
by maury91 on Tuesday December 26, 2017 @07:27AM
(#16008303)
This demo is done with my nearly 0 knowledge in lips reading, the effect I was searching was to make it more realistic, but with the limited amount of usable emojis I cannot have it very realistic, just better than random. At the end of the article I propose some gotchas to improve it: use better images, use a better map.
I'm happy that this article has triggered some people curiosity and can maybe evolve in something more advanced
Re:I made a talking emoji using regular emojis and JavaScript
by _nalply on Tuesday December 26, 2017 @07:44AM
(#16008358)
That's why I am bad at lipreading in English. People have mouth shapes all over the place. I'm sure I could learn lipreading a friend I meet often well, however this doesn't help lipreading a stranger. When traveling in the US I always have a notebook ready.
Re:I made a talking emoji using regular emojis and JavaScript
by scrollaway on Tuesday December 26, 2017 @06:11AM
(#16008107)
The entire IPA is available in unicode. You could treat those as ligatures with a special combinator character, which would work a lot like flags, skintones etc work. eg: [special begin character] + NEUTRAL FACE (U+1F610) + [IPA characters] + (special end character) = talking face! hurray!
Re:I made a talking emoji using regular emojis and JavaScript
by chrisseaton on Tuesday December 26, 2017 @07:18AM
(#16008279)
> Unicode isn't English or even Indo-European centric
It definitely is! The Latin alphabet gets a special place at the start of the range, in order to be compatible with ASCII, an English-language character set.
Re:I made a talking emoji using regular emojis and JavaScript
by throwanem on Tuesday December 26, 2017 @08:09AM
(#16008428)
That doesn't say Unicode is English-centric. That just says it sought backwards compatibility with a then-dominant character set that was. Seeing as otherwise Unicode would likely have been regarded as a pointless exercise in ocean-boiling in many more quarters than it initially was, and would have faced a commensurately steeper slope to reach wide adoption, I'm not sure what better alternative you see the initial designers having forgone to choose as they did.
Re:I made a talking emoji using regular emojis and JavaScript
by chrisseaton on Tuesday December 26, 2017 @08:30AM
(#16008497)
> I'm not sure what better alternative you see the initial designers having forgone to choose as they did
I didn't criticise it, did I? I just pointed it out. So I don't need to see any better alternative.
But I don't follow your argument - it's not English centric because they were only bending over backwards to make sure that the English-speaking world accepted it by designing it around them?
Re:I made a talking emoji using regular emojis and JavaScript
by throwanem on Tuesday December 26, 2017 @08:44AM
(#16008552)
Rather, because they chose to design it such that those regions of the world already not burdened with ugly character-set compatibility hacks (code pages! retch) would continue not to be so burdened, so as to ease adoption there and make it more likely they'd succeed in relieving everyone else of the compatibility hack burden. I'm not sure why it matters which regions those were.
Re:I made a talking emoji using regular emojis and JavaScript
by falsedan on Tuesday December 26, 2017 @07:09AM
(#16008254)
Only a few sounds use lip shapes. Most sounds are based on tongue positioning. Compare ‘m’ & ‘n’ to… not making any sound with your lips together or slightly parted.
The only sounds I would be very confident at recovering sounds from images for ‘th’, ‘w’, ‘o’, & indistinguishable voiced/unvoiced pairs like ‘b’/‘p’ or ‘v’/‘f’. Basically, the sounds a ventriloquist has to cheat.
Re:I made a talking emoji using regular emojis and JavaScript
by sametmax on Tuesday December 26, 2017 @10:34AM
(#16008988)
It may be true now, as modern block buster lipsync is atrocious (at least in France). But 20 years ago people cared a lot about it.
I remember distinctly this wonderful line from the 5th element:
- Are you classified as human?
- Negative. I'm a meat Popsicle!
Which they translated to french into:
- ...?
- Negatif. Je suis une mite en pull over !
Which has not relation at all to the initial line, and literally translateds mean "I'm a moth wearing a sweater".
But it's equally nonsensical, definitely as funny, and the beautiful thing is, the lipsync is near perfect.
However it's a lot of work, requires very talented people to achieve, and is mostly useless in the eyes of the right owners: money will come anyway if the movie is average, so why bother making it better ?
Re:I made a talking emoji using regular emojis and JavaScript
by maury91 on Tuesday December 26, 2017 @08:08AM
(#16008426)
That's false, I'm from Italy and our dubbing is made with big effort, and lips syncing is an important part, so important that sometimes they don't use literal translation, so they can choose words that have a similar pronunciation to the English ones and the lips look in sync
Re:I made a talking emoji using regular emojis and JavaScript
by exogen on Tuesday December 26, 2017 @05:15AM
(#16007987)
One improvement might be to use a pronunciation dictionary and map the emojis to actual sounds rather than characters. Having recently needed such a thing, I found that there are two widely available datasets: CMUdict and Moby Pronunciator:
The problem is that they'd be pretty hefty payloads to load on the client, so you'd want to do the text -> phone mapping elsewhere. Then use your character mapping as backup for words that aren't in the dictionaries.
Re:I made a talking emoji using regular emojis and JavaScript
by chrismorgan on Tuesday December 26, 2017 @07:06AM
(#16008248)
> an emoji, in reality, looks like this: “\xF0\x9F\x98\x81”
… well, that’s one encoding of the UTF-8 encoding of the Unicode code point.
> This is because setInterval(_=>{ },99) executes the function every 99ms
This is categorically wrong. You can’t trust setInterval to heed your request precisely at all: it’s a request that browsers take as a minimum only. Most browsers will call your function after the number of milliseconds you requested plus up to 16ms more, but some might wait even more than that (I think Safari in power saving mode doubles its tick time, to operate at 30fps; and background tabs typically won’t fire more than once a second these days).
Try running this snippet in your dev tools; it logs the number of milliseconds between calls:
On Firefox I’m getting mostly 108–111, with the odd one a little higher, and a couple of 99s after running it for a few minutes.
(Some trivia on similar techniques for this measurement: `+new Date` is rather slow, `Date.now()` is about 7× faster in at least Firefox and Chrome, and `performance.now()` gets you microsecond precision (it returns a floating-point number in milliseconds, tied to an unspecified epoch instead of real-world time), and is a little slower than Date.now().)
Also I neglected to mention other frame rates; 60fps is the most common, but you do get some 144fps devices, where you’ll be looking at about 7ms delays rather than 16ms.
Re:I made a talking emoji using regular emojis and JavaScript
by maury91 on Tuesday December 26, 2017 @07:18AM
(#16008280)
Hi Chris, thanks for the corrections, I know you cannot trust the browser, but for the sake of simplicity, and because the purpose of the intro was describing the code I found on the internet, I opt for a simple "this is the trick", doesn't always work, but this is the idea
Re:I made a talking emoji using regular emojis and JavaScript
by cornholio on Tuesday December 26, 2017 @09:14AM
(#16008665)
ycombinator can't:
There should be emoji after the colon, I can type them fine in Chrome. Truth be told, they would make up great return values, making yourself indispensable to your employer.
Re:I made a talking emoji using regular emojis and JavaScript
by throwanem on Tuesday December 26, 2017 @09:19AM
(#16008679)
The HN platform deliberately discards characters beyond IIRC 0xFF - it's been a while since I looked at the source, I'm probably wrong about where it cuts off. An odd design choice, but a design choice all the same - it doesn't bear on anything else in any way I can see.
Re:I made a talking emoji using regular emojis and JavaScript
by nitrogen on Tuesday December 26, 2017 @04:41PM
(#16011220)
There was a time when people discovered a lot of text renderers break down with lots of stacked combining characters, and people were posting strings to HN that IIRC either drew a stack of accents all the way up and down the page, or crashed the browser.
Maybe that has something to do with the character range limits. Not all non-ASCII is forbidden, though: Français should work AFAIK, and I've seen Chinese and Japanese.
Re:I made a talking emoji using regular emojis and JavaScript
by DiThi on Tuesday December 26, 2017 @05:38AM
(#16008039)
> Instead iterating through every byte like ""[n]
Pedantic note: JS iterates through UTF-16 2-byte code units. It's that way for compatibility reasons (used to be UCS-2), but it's the worst of both worlds: not as good as UTF-8 for ASCII, can't do operations in O(1) time like UTF-32.
Re:I made a talking emoji using regular emojis and JavaScript
by chrismorgan on Tuesday December 26, 2017 @06:53AM
(#16008209)
Calling it UTF-16 code units as distinct from UCS-2 is dubious. The water is very muddy there, because bad surrogates are allowed, thus the strings are arguably better considered UCS-2 rather than UTF-16.
I’m perpetually sad that a couple of big players decided to go all in on UCS-2/UTF-16 when it should already have been apparent that UTF-8 was roughly uniformly superior and that UCS-2/UTF-16 would have major issues. (I speak as one who was a child at the time, but has read a fair bit about the state of things back then. So don’t trust my yearning optimism.)
The worst part about UTF-16 is how it poisoned Unicode with surrogate pairs. UTF-32 and UTF-8 are both good because they didn’t need any special consideration from Unicode to make them work. But UTF-16, extending UCS-2 as it does… ugh.
Re:I made a talking emoji using regular emojis and JavaScript
by iverjo on Tuesday December 26, 2017 @07:07AM
(#16008252)
This idea originated in the jsgolf community and dwitter (a site where you post 140 character js snippets that generate interesting visuals). See https://www.dwitter.net/u/aemkei
Thanks for this extremely intriguing idea to use emoji to depict a talking person visually.
You know, some animated films get it right. Sometimes I clearly see utterances like «Thank you!», «Okay», but more often not. Especially older animations don't care and just let the character move the mouth in very simplistic ways: «Babbabbabaa Baababba ba baaa.»
Similarly for that emoji. It is moving the mouth in quite arbitrary ways. It looks like: «Sobbabbabee be <grin> seebee <grin> <frown> babbaa».
To solve this problem we need about twelve emoji for utterances: «ah», «ay», «ee», «oh», «oo», «b», «d», «f», «l», «n», «r» and «s». The «r» emoji must be animated (so we see the trilling tongue). The other sounds are either invisible like «k» or cover several different sounds like «b» also covering «m» and «p».
«Okay» would be rendered by: «oh», «oo», «oo» (standing for «k» wich is invisible) «ay» and «ee».
Edit: Add «th». This is an extremely simple sound to lipread. I remember my delight when I learnt English in my youth and in an movie suddenly realised that an actor said «Thank you».
p.s. Seems every day on here I'm reminding people from the US that their country is not the world, their version of English is spoken and spelt in just one country, etc etc etc. Maybe I should give up doing that.
The most classic maybe was a linguist (yes, amazing) who on her blog was gloating that her variety of US English was particularly rich, in having almost every vowel sound in English. I looked at her list.. it didn't include the 'o' sound in 'pot'..
I agree that the mouth shapes are garbage in this demo. A shame because that is the most interesting & important part of animating speech.
I'm happy that this article has triggered some people curiosity and can maybe evolve in something more advanced
*(just another rabbit hole actually, Unicode got plenty of them already)
"text engine developers hate him"
It definitely is! The Latin alphabet gets a special place at the start of the range, in order to be compatible with ASCII, an English-language character set.
I didn't criticise it, did I? I just pointed it out. So I don't need to see any better alternative.
But I don't follow your argument - it's not English centric because they were only bending over backwards to make sure that the English-speaking world accepted it by designing it around them?
The only sounds I would be very confident at recovering sounds from images for ‘th’, ‘w’, ‘o’, & indistinguishable voiced/unvoiced pairs like ‘b’/‘p’ or ‘v’/‘f’. Basically, the sounds a ventriloquist has to cheat.
Maybe that's why most animation studios don't bother.
Dubs. 90% of your viewers will see it in a foreign language, dubbed, where all your lipsyncing effort is useless anyway.
In fact, lipsyncing ut to english makes it look worse in the dub than having random nonsense movements.
I remember distinctly this wonderful line from the 5th element:
- Are you classified as human?
- Negative. I'm a meat Popsicle!
Which they translated to french into:
- ...?
- Negatif. Je suis une mite en pull over !
Which has not relation at all to the initial line, and literally translateds mean "I'm a moth wearing a sweater".
But it's equally nonsensical, definitely as funny, and the beautiful thing is, the lipsync is near perfect.
However it's a lot of work, requires very talented people to achieve, and is mostly useless in the eyes of the right owners: money will come anyway if the movie is average, so why bother making it better ?
As child (due to deafness until I was almost 5) I relied a lot on lip reading, and it was very noticeable if something was dubbed.
http://onlinebooks.library.upenn.edu/webbin/gutbook/lookup?n...
https://github.com/cmusphinx/cmudict
The problem is that they'd be pretty hefty payloads to load on the client, so you'd want to do the text -> phone mapping elsewhere. Then use your character mapping as backup for words that aren't in the dictionaries.
… well, that’s one encoding of the UTF-8 encoding of the Unicode code point.
> This is because setInterval(_=>{ },99) executes the function every 99ms
This is categorically wrong. You can’t trust setInterval to heed your request precisely at all: it’s a request that browsers take as a minimum only. Most browsers will call your function after the number of milliseconds you requested plus up to 16ms more, but some might wait even more than that (I think Safari in power saving mode doubles its tick time, to operate at 30fps; and background tabs typically won’t fire more than once a second these days).
Try running this snippet in your dev tools; it logs the number of milliseconds between calls:
On Firefox I’m getting mostly 108–111, with the odd one a little higher, and a couple of 99s after running it for a few minutes.(Some trivia on similar techniques for this measurement: `+new Date` is rather slow, `Date.now()` is about 7× faster in at least Firefox and Chrome, and `performance.now()` gets you microsecond precision (it returns a floating-point number in milliseconds, tied to an unspecified epoch instead of real-world time), and is a little slower than Date.now().)
Also I neglected to mention other frame rates; 60fps is the most common, but you do get some 144fps devices, where you’ll be looking at about 7ms delays rather than 16ms.
99 % 6 === 3 ((99 % 6) + 99) % 6 === 0 ((((99 % 6) + 99) % 6) + 99) % 6 === 3
[0]: https://en.wikipedia.org/wiki/Microsoft_Comic_Chat
There should be emoji after the colon, I can type them fine in Chrome. Truth be told, they would make up great return values, making yourself indispensable to your employer.
Maybe that has something to do with the character range limits. Not all non-ASCII is forbidden, though: Français should work AFAIK, and I've seen Chinese and Japanese.
Pedantic note: JS iterates through UTF-16 2-byte code units. It's that way for compatibility reasons (used to be UCS-2), but it's the worst of both worlds: not as good as UTF-8 for ASCII, can't do operations in O(1) time like UTF-32.
See https://mathiasbynens.be/notes/javascript-encoding for a discussion of the topic.
I’m perpetually sad that a couple of big players decided to go all in on UCS-2/UTF-16 when it should already have been apparent that UTF-8 was roughly uniformly superior and that UCS-2/UTF-16 would have major issues. (I speak as one who was a child at the time, but has read a fair bit about the state of things back then. So don’t trust my yearning optimism.)
The worst part about UTF-16 is how it poisoned Unicode with surrogate pairs. UTF-32 and UTF-8 are both good because they didn’t need any special consideration from Unicode to make them work. But UTF-16, extending UCS-2 as it does… ugh.