"Known-good" sources have a serious habit of getting owned by malware and replaced by organized crime schemes. Anti-abuse can't be handled by whitelists, it needs to be an online system.

Garbage sources have a serious habit of getting taken over. It's not that common with quality sites, but also not impossible. A simple downvoting system would be good enough.

The problem is that even quality sites die. And when they do, their domain gets poached and suddenly it's not a quality site anymore. But according to the ranking algorithms it's the same site, since its quality is measured by how many links it gets.

Coopting quality sites is a bit of a cottage industry. There is also a significant amount of hacked wordpress sites. Their domain may be quite reputable, but they're unknowingly host to a ton of spam.