Of course it would be better if the "bully" was removed or reformed.

However, until that happens (which might be never), it's prudent to protect yourself, and telling people that they must take precautions for their own safety and well-being is not "blaming the victim".