Why do you believe he gave you the real algorithm?

Access to the piece of paper severely reduces the attack space, even if it isn't "go diagonal from (letter2, letter3) of domain name". Heck, it is not much different if it is a trivial transform on the characters themselves. Anything that a person could easily do (making the thing convenient enough to actually use) is not going significantly alter the number of permutations. Of course, this is making a big assumption that a targeted attack will go after the paper rather than say, the keyboard (which is generally much less secure than a wallet.).

However, this being said, it is much more secure against untargeted attacks than the standard "i have an algorithms in my head" or "i use the same password" approach a lot of people use.

Once you're in the realm of targeted attacks you're in a whole different ball game. You have to consider who the attacker is likely to be and what they have access to. If your wallet isn't secure enough you can upgrade to a fire safe or some such. That narrows it do to people who you let in your home or are willing to break in. Beyond that though you're starting to enter into the realm of James Bond shit.

Or into the realm of two-factor auth. Sometimes I wonder why my roommate can get a security token for his World of Warcraft account, and I can't get one for my bank...