Some people do not know they are in one of those groups, and ARE likely to experience harm. These groups (unaware immune issues) are statistically insignificant compared to the entire population, of course.
But ignoring them is in fact contributing to the antivax movement; primes are statistically rare but if you told people they don’t exist, you would be working for the antimath movement.
The problem is: Even to well educated parents the summary "Everyone could be affected by life-long effects due to vaccination" has a drastically different meaning compared to an objective bystander that looks at the real numbers, and this leads to (seemingly irrational) decisions. Some even are mislead into feeding crusades against this "well known danger called vaccination".
Something like 1 in million do have a serious side effect of the vaccine. This is far less than the 6000 dead in Congo because they don't have the vaccine, but it isn't zero. The fact that it is far less is reason to vaccinate anyway - but if doctors can determine that your child is one of that 1 in a million before hand (they normally cannot!) it would be wise to skip vaccination for just those kids.