1. Are all your passwords unique?
2. If I discovered some of your passwords, will the rest of your passwords stay secure?
3. Were all your passwords created using at least 32 bits of entropy?
4. Are your passwords stored only in encrypted form?
5. Do you perfectly remember every single password you’ve used when signing up?
6. Do you turn up positive for a password leak at this website?
If you answer "no" to any of these questions, you'll benefit from a password manager.
About storing passwords, I use LastPass and they use client-side encryption[1], which means even they don't have the decryption key to read my passwords. So, you'll be fine as long as you have a secure passphrase and 2FA :)
https://lastpass.com/support.php?cmd=showfaq&id=6926
[1]: Please skip to "Strategy No. 1: Proper Password Management" @ https://www.toptal.com/remote/best-security-practices-for-re...
And nobody hacks into your machine...
EDITED to add: I am using Password Safe which is recommended by Bruce Schneier. What you describe would be an absolute noob mistake. He would be pretty embarrassed if you were right.
Even if in theorie they are safe. Even the slight chance that a single failure could lead to all my passwords getting in the wrong hands at once just is to scary.