I agree. At best the argument that you can trust "smart" people is circular: that smartness is a socially constructed measure of how "good/right/clever" everyone else finds your thinking. so in general people will endorse the opinions of those they endorse as "smart people"