1. I would 100% forget which symbol goes with which site.
2. The length I choose for my password may be too long/short for a given site.
3. Some sites require special symbols; some sites forbid them. Now I have to pick out my symbol based on the site's requirements, not ease of memorization.
I could use a password manager but I don't want to be dependent on one piece of software, there's too many failure modes. It's already bad enough that all my accounts share a limited set of email addresses.
The main issue I have with schemes like this is that there's no repository of global identifiers for websites and services. I can build a password for blizzard.com, starting from say "bl", then I forget about it and five years later their website is now activision.com, and I wonder why I can't log in with a password built from "ac". It's a minor issue since password resets are a thing and rebrands are rare but still..
http://www.passwordcard.org/en