> I would imagine Apple doing this with FaceTime soon.

What's this claim based on? Last I looked into FaceTime tech, they didn't do anything special - their quality comes from use of H.265 and the fact that iOS devices have good quality HW encoding blocks which provide good compression at low bandwidths.

FaceTime stream is usually also low motion / change so it's possible to achieve very good compression even with basic quality. Although they still don't quite match the quality of AV1 powered Google Duo on very poor connections.

I'm not the person you're replying to, but I think the main reason you'd expect to see this on iOS is due to the additional Face ID sensors available on most of their devices. Presumably Apple could offer a version of this tech that features higher quality tracking and thus a more realistic and natural-looking final output.

The latest NPU can't deliver this kind of "deep compression" at 25fps, not even 10fps. But in the future they can just send the Facemesh vertices and streaming text of the speech (and classically compress non-speech audio, if it's even desired as most people are happy just using it to talk) so it will be less data than 1Kbps.

I think Face ID could be used to create the point map, instead of generating it from an image. They could also use Face ID to prevent the tech from malicious deep fakes, e.g only allow people to use this feature I when Face ID confirms the user and the manipulated photo are the same person.

Isn’t the point map supposedly encrypted on the Secure Enclave?[a] If it is, you can’t access it from the main CPU; you can only ask it to compare a provided one with the stored one.

[a]: I know the password is stored on it, but idk about the face mesh

The verifier point cloud used for unlocking the phone must be kept in the secure element.

That doesn’t mean you can’t read a point cloud for other purposes.

They currently expose ARPointCloud API but I don’t know what sensors it uses to produce it;

https://twitter.com/nobbis/status/1292262455490629633?lang=e...

> Remember when the industry were trying to compress CD Audio quality (aka 128Kbps MP3) down to 64Kbps?

128kb mp3 is good enough for most people most of the time, but it isn't CD quality. Having said that, 64kb Opus is almost or about as good as 128kb mp3.

I wonder how well these techniques can be applied to audio.

I doubt the general technique described here would have any applicability to audio, unless the idea is to dynamically create a text-to-speech model of the speaker and transmit only text. I don’t see that being practical any time soon.

> Not so sure where Nvidia is heading for this since Not everyone will have a CUDA GPU.

Maybe it will be exclusive to Android devices.

Or maybe it will work on any device (consuming CPU or GPU depending on the hardware) but only on Nvidia's communication app.

> I would imagine Apple doing this with FaceTime soon.

And we'll notice it because somehow our and other people faces in facetime will start to subtly translate strangely aggravating emotions, as current memojis do.

The problem is with phone the background change more, and you show more things than just a static face, because it is easier to move around.

Probably just turning an R&D project into a marketing feature.