I think anti-vaxxer is painted with a much wider brush than that, and somehow become automatically equivalent to "anti-science"; there is overlap but there's also a lot of non-overlap. Somehow vaccines have become an "all or nothing" proposition which is just as stupid as saying that all medications are created equal.

But in any case, I'm not going to live my life by what labels others put on me.

That's the rational response, but in practice what we saw is that anyone who even questioned the Covid vaccine was instantly labeled anti-vax. It didn't matter if they were fine with all other vaccines. It didn't matter if they just had Covid. Any resistence was shut down as anti-vax.