Well as I said, the majority of DNS servers are meant for a specific group of users, of a particular ISP or company, which can be filtered by IP. Google & OpenDNS are exceptions; I wouldn't know about their defense mechanisms and I'd imagine they prefer not to disclose them, but I suppose it might be similar to how credit card companies try to detect fraud by looking for unusual transactions. There's also non-recursive servers answering queries about a particular domain, they also answer to anyone but they answer only about their domains (authoritative).

I think your point about trying to find a way for the open DNS resolvers to be protected but still open is moot, because there's really no reason for servers to be open to everyone (except for Google or OpenDNS who know what they're doing), these are just misconfigured servers of people who have no idea they can be abused. In my opinion the solution is for ISPs to actively block such open resolvers among their clients.