There are two things a vaccine can do - protect you, or protect society. Their greatest benefit is when they protect society by dampening the spread/severity of a pathogen to a level low enough that society can function essentially normally.

If a sizeable chunk of the population chooses not to participate in that collective action, they endanger themselves (which can be argued is their fair choice), but also those around them who are unable to take the vaccine. There is no individualist refusal to vaccinate, the effect is still collective, just in a negative direction.