Storing your keys in secure hardware on a phone is almost certainly more secure than storing a key in software on your desktop hard drive.

If you don't trust your hardware, it's almost always game over. Desktops have devices running dubious firwmare as well, but at least with a hardware key store, the window of compromise ends at the time you update – a stolen key stays stolen forever.