DotA2 actually has some measures in place that lessen the impact of trolls or AKF'ing/quitting players, such as sharing their gold and control among the remaining team, or disabling their abilities that can harm their own allies. In League of Legends, it's a guaranteed defeat if one ally decides to troll or quit (assuming comparable skill on both teams.)

I've always thought that instead of reports and bans, team games should have a peer reputation system, like Reddit or HN's voting.*

MMR/game skills should not be the only criteria for matchmaking; it should include reputation/social skills as well. Oh and disallow global chat in ranked matches.

EDIT: Yes, it makes for a nice thought experiment to come up with a reputation system that would be resistant to "gaming" or abuse, as others have mentioned.

One possibility is to associate a cost to upvoting, and/or make downvoting an earned privilege, similar to how new accounts on HN cannot downvote.