in Tor servers are one IP and one public key, so yeah you can't talk to that ip if they change their public key. It's closely related. But this is because someone is giving you a table of that ip <-> public key association. If an attacker change the public key of an ip in that table then... That's why OP's idea is stronger.