The original search engines ranked pages by their content. Naturally this led to gaming by including keywords (remember huge invisible sections of pages that just repeated keywords thousands of times?).
Google's original PageRank algorithm was a complete breakthrough for this, almost completely disregarding content and instead ranking results based on the text other pages had used to link to the page. This was so good, in fact, that most of the other search engines from the time didn't survive.
This once again led to large-scale gaming with techniques like link farms, and a small arms race as Google came up with ways to squash these new techniques. The quasi-legitimate "SEO" industry spun out of this.
I think now we're in the same cycle again, and at this moment scam sites are winning. What's to be seen is if there's a looming breakthrough, or the arms race will continue.
This is likely because marc.info isn't playing the SEO game, and spamsite #43254 is.
Once upon a time it was possible to identify "good" pages by how often they were linked - effectively crowd-sourcing the problem.
Natural selection: sites have learned to manipulate the game and the crowd.
Under pressure to optimize metrics that lead to SEO and better valuations, the internet is getting less useful from a user perspective.
I don't want to watch a video/slideshow, download an app, register for a forum, or read through a 2000 word fluff piece with interspersed ads and links to more information in which site has buried the one-sentence answer to a 2-second question in order to maximize my time on the site.
This garbage is what makes it to the front page of google these days. I guess the poor sites that aren't user hostile just aren't SEO'd enough.