And even if they didn't, I wouldn't consider hackers getting my bank info-- or governments reading my email-- the server operator's problem.
It certainly wouldn't be worth setting up a MiTM attack between my house and the first Comcast router my cable hits. That approach doesn't scale. Sure maybe if I'm Warren Buffet (or some mobster if we're talking about the government) it might make sense. And if they did do that, then NO source I connect to is a trusted channel, as they could MiTM any and all connections.
The scheme wouldn't (always) protect you from an attacker setup somewhere in between you and your destination, like the NSA servers at AT&T, or the Firewall of China. Your trusted source could hit the same bad path.
And by your own acknowledgment, the scheme doesn't protect you from a hacker breaking into a bank's data center. If a hacker is smart, and only grabs say every 100,000th credit card, how long will it take to isolate the location of the exploit?
So the scheme doesn't really protect against anything with any level of confidence.
Sure, the scheme is only as good as your trusted source. But the trusted source could be very good. It could have multiple servers in different ISPs and even different countries verifying certs for you, and could use other methods of verification as well.
The bank can easily check if their connections are getting MITM attacked. The hacker has to modify the certs seen by anyone who contacts the bank, even if he doesn't MITM every connection, and that would be easy for the bank to detect.
What if you add that information to a key continuity system? I feel like this is an obvious idea, but I've never heard of a system using it. It's simple: when you first see a certificate you send it to a trusted third party to check. That third party doesn't need to be or even know about the CA who originally issued the cert; all it has to do is make its own connection to the server and make sure it gets the same cert you did. That protects against any MITM on your end of the connection.
To succeed, a MITM would have to control either the server's connection (which becomes the server operator's problem), or both your connection and the trusted third party's connection (which becomes the trusted third party's problem). I'd rather trust a single third party of my choosing than a huge list of CAs chosen by an opaque and somewhat arbitrary process.