It's not just that. I generally avoid taking medications unless I need to (when prescribed), which so far in my life has been extremely rare. It may not be "harmful", but I don't want to put any chemicals in my body if I don't have to. Most medications come at some cost, in that besides doing what they are designed to do, they also interact with other cells or systems in ways that may not be desirable. Vaccines are no different in that respect.