Yes, words are polysemic. In this case, 'anti-semetic' is seen as "anti-jewish". That's one sense of 'anti-semetic'. You can also add another sense of that word: "against speakers of Semetic language or people". This sense is not used when people use 'anti-semetic'.
In Europe, there has been centuries of discussion about jews: it was called "the Jewish question". Anyway, this anti-semitism started as a product of religious rivalry (which religion is True? Christianity or Judaism). Then extra features were added, etc.
Maybe because that hatred is more widespread than hatred against all Semitic people...?
The fact that Israel is a nation "home for jews" doesn't help, as some people think that every critic against Israel politic is the mark of "anti-semitism" even if religion has nothing to do with. Sometimes it might be the indicator of "anti-sionism" and sometimes... just the indicator of political difference! After all, you can criticize any country for its politic without critizing the counrty "per se"
Moreover, a lot of people in Israel are not jewish but muslims.
[1] https://en.wikipedia.org/wiki/Semitic_people