Imagine the question : "What is the best templating engine to use for Django?" This question doesn't have an answer, because it's so dependent on use case (even that statement is controversial). So if it's on SO and not closed, you'll get a bunch of answers that are contradictory, and people flaming in the comments.
I think the main objective for the SO team is to have people land on the site and _have their answer right there_. By removing possibilities of these off-topic questions, you're less likely to land on answers that help your specific thing.
End result is "when I land on SO I get an answer every time" instead of "when I land on SO sometimes I get a flame war". For branding, at least, I think it works in their favor to close those questions
e.g. http://stackoverflow.com/questions/257712/best-web-applicati...
or less seriously:
http://stackoverflow.com/questions/184618/what-is-the-best-c...
Also, smart, general questions are not the only questions to be closed: I got one closed on networking because the problem was between two company networks, not inside. There is an obvious reason for a rule about company networks and that is to keep hone networking out, not because we need another subsection for WANs, no?
It's unbelievably annoying when this happens. I have a number of bookmarks to incredibly insightful posts, whose usefulness has been corroborated by literally hundreds of people...all closed because they are deemed "off-topic" or some such by an admin.
My theory is that SO was essentially "done" years ago. But people are still on staff and needed something to do. Thus, unrelenting navel gazing ensued.