If I'm verifying the fingerprint of the server's certificate, is that not doing the same thing that the chain of trust would be doing?
Nobody is really verifying the fingerprint of the server's certificate, but if you are, fine; you and I are talking about two different issues. You're talking about your own homebrew setup that uses the SSL protocol and manual certificate verification (incidentally, just set up your own CA and forget about the fingerprints; it's not hard). I'm talking about Internet-scale secure web connections to banks.
People on HN can and do argue about this until they are blue in the face, but the fact of the matter is that authenticated connections aren't just a special "bonus" that SSL provides beyond encryption. Every secure encrypted protocol makes arrangements to authenticate keys; this is the same thing that makes your first SSH connection to a new server insecure (demonstrating that problem used to be a sport at security conventions).