Getting better recommendations was even presented as the reason why you should upvote/downvote posts.
But recommendation engines are very hard. Just look how poorly Amazon's work, despite its central role in corralling gigantic amounts of cash, the manpower and the raw talent that Amazon possesses. Reddit's smart customized home page wasn't better than the generic one. Eventually, multiplying subreddits and letting people [un]subscribe to them solved the problem better than going all AI.
As for fighting voting brigades, I believe they continuously work very hard on this, but rely on secrecy to keep their measures effective.
As for Amazon, maybe its recommendations are intentionally unrefined, like how Target discovered it needed to obscure its direct mail campaigns aimed at newly pregnant women ( http://www.nytimes.com/2012/02/19/magazine/shopping-habits.h... ). Granted, it's plausible Amazon just hasn't cracked the code yet.
But I use reddit mostly to follow news about my favorite sports teams and a few of my hobbies. For that, I don't care what the rest of reddit is saying.
In fact, I suspect allowing more isolation between factions would allow for a diversity of opinions to brew in those pockets, with the good ideas inevitably metastasizing to the whole network.
Rather than the surfaced comments being a function of raw upvote count, the upvotes could count for or against you seeing the comment based on whether they came from someone who tends to deliver you read-worthy content. Racists would see all the racist comments they want; others not so much.
This could also decrease the influence of voting brigades and others trying to game the system.