The weak point is you still don't know what you're signing, who or what you're authenticating, etc. if you don't have a trusted display and do the entire transaction on trusted hardware. (your host may be compromised, and could show arbitrary data, proxy connections, etc.) Protecting the key does go a long way, but even with a secure passphrase input and smartcard key operations, there is risk.

The ideal, IMO, would be some secondary hardware with a limited, secure OS which lets you put specific limits on how the key is used, showing a hash or identifying details of each transaction on the secure display, prompting you to verify each. E.g. to send a payment, the amount and payee (and date/sequence number) are shown on the secure display to verify before authorizing.