I agree, and my take is that the mass of people has, between them, much more "domain wisdom". Different groups have deep understanding of different things. Villagers understand small settlement dynamics; urban dwellers understand towns. People near the sea understand all things related to it; people in grasslands understand nuances of farming better than those in the mountains, etc.

The problem is, you have to reach relevant subgroups to access that wisdom - otherwise, when you're just polling the entire population on a specific topic, well... few subgroups are experts, but everyone has an opinion, so it averages down to "dumb, panicky animals".