That'd be less secure than a completely separate device with your authentication data on it, but no less secure than current-generation password managers as far as I can tell.
Even the whole "use a cell phone for auth" is not really a good idea. It at least sort of qualifies as "something you have", but a great deal of the identity that you "have" isn't in the physical hardware, but in the cloud and the connection the device has to the cell network, which as "something you have" goes is pretty low-grade security.
You and thedufer aren't doing this right. You see one objection, then bend the system to meet it, then see another objection, then rewrite the system to meet that, then so on, never considering the whole picture. You can't do security that way. You have to do it holistically. The whole system that you come up with has to be simultaneously secure against everything, and also needs to be the best possible solution. If you're looping a design around trying to solve one problem at a time (and not necessarily all that well) you've already lost.
As a result, this design inherently segments out the login to only the upper tier of affluent users. Depending on your application, this is either a huge win or a huge barrier. All about your context. If this fits your target user based awesome. Eventually, though, a normal email-password solution may be needed if you grow your market out to the more general public.
Still, a promising idea. Just go in with your eyes open to the tradeoffs.