When a spammer begins to spam on a certain channel, a message can be sent via a secondary channel (maybe a special control channel, or just any combination of other known channels that hits everyone who had authorized access to the original channel) to automatically shift just the valid users over to a new channel, and discard the original. It could be completely invisible from a UI perspective, with the new channel retaining all the old channel's metadata.
The nice thing is that it's almost entirely compatible with all the e-mail infrastructure we have now, but also allows for a very convenient use-case if you're willing to switch to a new UI. The only problem is in people's current knowledge of, and access to, public-key encryption. If Facebook or another major identity provider started pushing people to send them public keys and set up a queriable API for them, that'd be a nice first step toward solving that.
And if a potential client can, why can a spammer not?
However, assuming ubiquitous identity servers, of the kind Diaspora is trying to enable, each identity server could be subscribed to a feed of updates for a respective channel key. So, for example, you could have an identity server at work from which other employees could pull your current "work" channel key, or an identity server at a hackathon meet-up from which other people who came to the meet-up could pull your current "meet-up" channel key. Then, people who contacted you would be contacting you with the trust score of the identity server pre-applied (you'd assign your work identity server a high score, because only employees may be members, while the meet-up score would be lower because anyone could show up and sign in, etc.) If you only want to deal with people you've physically met, you can just draw a line saying to discard messages below a certain trust score, and be done with it.
If not, though, channels are still a helpful complement to spam filtering, as each channel could have its own spam filter trained on channel content—so something that would be perfectly fine in a "work.offtopic" channel would be considered spam in a "work.ontopic" channel. Because people would likely have multiple means of contacting them (you'd have access to most people you actually knew through multiple channels), individual channels could be much harsher about rejecting content—probably just silently discarding it for everyone but the sender, and either acknowledging the rejection to the sender, or just saying nothing. (If clients sent a message to all relevant channels simultaneously, and messages from different channels with the same ID were re-combined for other clients, this wouldn't be so much of a problem.)
Any extension to this concept would simply be sticking an identity server in a new place: one sitting on each person's phone or computer providing temporary access to a "proximity channel" that constantly shifts (so as soon as you go out of range, you're unsubscribed from the identity server and can no longer send messages), topic-oriented free-to-join identity servers (basically equivalent to IRC servers, except that they don't have to handle message routing themselves), and so on.
Then you get into the spam-filtering based on content, which is already successful at preventing massive amounts of spam.
So I don't really understand how what you're proposing makes a difference.