I think you missed my point: phones can be stolen. If you watch me type in my passcode by looking over my shoulder on the train (knowledge factor is now compromised), then steal the thing and run off as the train is departing, then you have full access.

Go ahead and steal my RSA token, and see what good it does you on its own. None, right? Not unless you bludgeoned my password out of me first.

All of the "have" factors that I've seen are, fundamentally, just a big number that's hard to get at or copy (with the exception of a physical, metal key). There's some math done on that number to generate a one-time-use code and do it in such a way that you still can't access the big number, but if an attacker were to compromise the number, then yes, that factor is broken. The theory is that the device cannot leak the number, and therefore it will only ever be known by the authenticating party. Or, more accurately, that the knowledge contained within should only ever be able to exist in two physical places (one each with the party being authenticated and the party performing the authentication).

I'm guessing we're talking about slightly different things and use-cases, but it's impractical to know that without a lot of diagramming and back-and-forth.

Yeah, it's fair to say the unlock passcode on your phone is more vulnerable than a specific authentication password used during a web login. But other than length and how frequently it's exposed, it's not fundamentally different from shoulder-surfing that password and then stealing your keyfob. In both cases you can try to deactivate the account before the next login.

Biometrics are probably the best way to prove identity and control of your phone, locally. They already have a high-quality CCD. Biometrics are unpopular as something sent to remote devices for a variety of reasons, but should be much more palatable if they remain local-only.