The word you are looking for is concerned not affected. As you yourself pointed out, so far he hasn't done anything that actually affects you. If you want people to take you seriously, you need to not play the victim card and instead make rational arguments.