I agree, but it's not even that.

I don't think most people really try to argue against "science" as is, or even the scientific process in the large scale. The gripe we have is against people who first do bad science (or popularize them), refuse to admit they're wrong, then patronizingly tell the public to trust them anyways, and still have the audacity to reassure us that "it will all work out (50 years later)".