But just the software-based ways that I could potentially be stolen from are too plausible to ignore. I have never felt that I could say with total certainty that my system is not compromised or vulnerable to compromise. Even a bug written with no malice could be my undoing. Working in the software industry has given me very little faith in the infallibility of software, both open and closed source.
Of course many people hold large quantities of crypto and are fine. But a few have it lost or stolen. Even super simple attacks sometimes work, like the malware that would poll the system clipboard waiting for it to contain for a BTC public key, and if found, swap it with one owned by the virus-writer -- so that you inadvertently paste the thief's key in as the destination for the transfer you're about to initiate. When I first heard about that attack, I thought it was clever and had to really think: hmm, would I have fallen for that? Usually I double check, but if they were smart enough to generate a large set of wallet public keys so they could replace the destination with one that starts with the same few characters, I probably could've been duped at some moment of carelessness.
* The Bitcoin client software I downloaded (I didn't build from source, and even if I did, I wouldn't have read the entire source first, and even if I did, I could get Ken Thompson'd)
* A computer virus delivered in a game I install, or hardware drivers I update, that would be virtually undetectable until it's too late
* A fire that destroys my home / somebody compromising off-site backup of my private key (pick one or the other risk)
* An untrustworthy exchange or an exchange that is unfortunately compromised at the moment I choose to transform them into Real Money(tm) -- while I kept my coins in a wallet off an exchange, I would still need to turn them into USD someday since the dream of paying for ordinary things directly with Bitcoin is wheezing on life support
Some of those concerns can be mitigated with a hardware wallet, but not all, and you still must have faith in the people who designed the hardware wallet and the manufacturer who actually built it.
At least with traditional banking, if my trust is violated I usually have some recourse by which I can recover my money. E.g. if my PC is hacked and the hacker transfers money from my bank account, as long as I see that happen within a few days I can probably have the bank stop it and get my money back. With crypto one false move and you lose your stuff, do not pass go.
In short, the "trustlessness" protects me from a pretty low-risk problem -- I would say practically a hypothetical problem at least in the U.S. -- but exposes me to a high-risk problem.