I realize that now. One way to mitigate it would be to have the browser somehow tie a given client certificate with a particular website. That is, the client cert for news.ycombinator.com, would only be presented if I try to connect with that server and nothing else.

That way, if I go to a phishing website that pretended to be Hacker News, my client certificate would not be sent and my browser could warn me by saying that the connection is not using a client certificate. Right now, if we only rely on server side certificates, there's nothing stopping a phishing website from using Let's Encrypt to show the secure connection icon in the URL bar and tricking me into thinking it's a legitimate server.