"Vaxxers", as you have so put it, (or as I like to call them, normal human beings) have reached a level of proof that vaccines are safe, and that, specifically, the COVID-19 vaccines currently in deployment are several magnitudes safer than, say, contracting COVID-19 itself.
"Anti-Vaxxers", however, have (un?)intentionally proved that vaccines work, performing one of the largest voluntary human drug trials in history as the placebo group. Their sacrifice shall, hopefully, not be forgotten (lest we repeat it).
No one has a right to harm another person. Knowingly transmitting an infectious disease, after being repeatedly informed that it is, indeed, an infectious disease, and that the victims, worldwide, total almost 5 million worldwide and continues to climb, and they still continue to spread it, that is what makes someone a terrorist: you harm, maim, and kill people to spread discord. Why you do it is immaterial, "I didn't know", "I didn't understand", "I was following orders", are not excuses in a court of law.
Although I do agree with you that anti-vaxers should be treated no different than common terrorists; as Americans, under the law, they have a right to declare that they are terrorists and give their little illogical terrorist rants. The First Amendment is very clear on what is not covered, and the courts have repeatedly confirmed that being wrong, being disingenuous, and lying is covered (as long as you are not committing perjury or other similar, actual, crimes).
Our founding fathers knew exactly what they were doing when they penned this: in their day, they also had their form of denialism. The first amendment is, essentially, "It is better to keep silent and be thought a fool, than to speak and remove all doubts.", but weaponized against the idiots that shall forever plague us.
That said, if a rational adult, one that we have, as a society, cannot tell the difference between the truth and the lies, then we have both failed as a society, but also have trusted an adult to actually act like one and they, personally, made the choice to act like a spoiled brat; acting like a spoiled brat is not a victimless crime, and sometimes, but not always, they are punished like an adult for violating the trust of the society that they live in.