I think some countries (such as the UK) recognize religious and ethical reasons. For example you can say that you're a vegan and can't take the vaccine because they're tested on animals.

But I'd rather be injected with a litre of smallpox than go vegan.