Scientists say it is much worse than the flu. For the vast majority who actually experience the disease, it is not much worse than the flu. Thus, people start to mistrust the science because there's a disparity between the statistics on paper and what people actually experience in reality. I think most people understand now that covid is basically like the flu 99% of the time, and much worse than the flu for a tiny subset (and for the most part, the tiny subset can self-identify).
You're probably right: it's hard to understand the difference between statistical evidence and anecdotes. On the other hand, a lot of other people who don't experience COVID still seem to be able to separate their own lack of statistical knowledge from the ability to trust scientists. I think these mask deniers are mistrusting scientists for some other reason.
[0] https://www.cdc.gov/coronavirus/2019-ncov/covid-data/covidvi...
If you'd rather believe Fox News than scientists who are telling you this is much worse than the flu, then while you might not be a bad person, you've earned my mistrust.
You can drop your mask if you want, but you can't really blame the OP for not trusting you.