Pretty cool paper: they train LDA model to predict password from sound of taps with 61% accuracy! This required background app running on device with access to microphone. The obvious extension is to predict password from video clip of someone entering it!

The front camera has very little chance to register any finger motion on screen, though.

Adding accelerometer (tilt / shift) data could help a little bit, because it's correlated to the motion of typing hands, too.

It helps quite a bit. The relevant web APIs were retroactively rate-limitied to mitigate these risks.

https://bugs.chromium.org/p/chromium/issues/detail?id=421691

https://arxiv.org/abs/1410.7746

There were 3 research papers that came out around 2012 (almost simultaneously) that looked at using accelerometers to infer what people were typing.

Here is one of them, I don't recall the other two. https://dl.acm.org/citation.cfm?id=2162095

It could look at your eyes to infer wich key you are pressing

Honestly, I'd really prefer that to all the times when I have slightly damp fingers an my fingerprint sensor is not registering, or it is causing issues with swiping.

For PIN codes, would randomizing the placement of the keys on the virtual 10-key keyboard mitigate this?

Yes, and it’d help with simple finger oil on screen analysis, but it’d make for a frustrating ux.

LineageOS has this as an option.

It's actually not so bad UX-wise.

So does using an alphanumeric passcode instead. Especially if you include capital letters and punctuation that you can access with a swipe starting at the shift or number key (so you never actually tap the key that produces the input).

ICBC does this on the full keyboard, but they randomize everytime you switch from symbols <-> normal keyboard. Makes for a frustrating UX.

Could be cool to combine with an accelerometer and have an accurate keyboard with no moving parts and without multitouch. Could be quite rugged.

Yet another reason why the mobile OS should (freeze / swap out / disable) an app upon entering the background ... at a minimum the user needs to have more control over degree to which above happens ... of course powers to be love a smartphones current total lack of such privacy

I wonder how much accuracy you could get doing the same thing with the IMU.

> In controlled settings, our prediction model can on average classify the PIN entered 43% of the time and pattern 73% of the time within 5 attempts when selecting from a test set of 50 PINs and 50 patterns. In uncontrolled settings, while users are walking, our model can still classify 20% of the PINs and 40% of the patterns within 5 attempts.

https://www.cs.swarthmore.edu/~aviv/papers/aviv-acsac12-acce...

Brings a new meaning to touch typing.