And your wife bought something from my eBay store. Now I have your home address.
And if I am a ruthless character then I quietly break into your house one day with th3e objective of leaving no sign I was ever there. Search for written down passwords, take a photo, leave.
You'd find a legit way into the house.
The point is that companies put vast effort into digital security but in many cases it's easily compromised by going to the home of the person that is the hacking target.
Especially if your parents accidentally left you home alone..
The perfect password book is combined with a word you remember but don't write down as a pepper, but I doubt it's much of a problem in practice; it takes one leak of an u hashed password to break the code.
I think for many the risk of someone breaking in and stealing your password book is much smaller than the risk of a centralised password manager getting hacked (LastPass and friends).
On a desk at home? It is marginal, certainly a burglary is a low frequency event, but we also have events like fire that make it insecure in other ways.