When you're debating with someone about how the Higgs Field does or doesn't imply the existence of fundamental particles, or how neutrinos do or don't move faster than light, then yes, appeal to scientific authority is "troubling" as you mention.

But when we are talking about vaccines, flat earthers, etc, then how can you possibly justify that it's bad for

>people are castigated for not believing in scientific "truths".

?

Perhaps flat earthers is a silly but harmless phenomenon, but there is real pain and suffering from other kinds of science denial such as the antivaxxers.