Hungarian isn't Slavic.

The word is probably borrowed into Hungarian from Slavic. We have a similar word in our language as well, but it's used as someone who doesn't understand. Also used interchangeably with Turk.

Do you mean Barbar?

No, it's very similar to nemec, basically the same thing also borrowed from Slavic. The "c" ending is pronounced as "ts". The full phrase goes like: "Why don't you understand, are you German/Turk?" (as in not understanding the language). There's also a place, a citadel and river with that name, which your ancestors might have been familiar with:

https://en.wikipedia.org/wiki/Siege_of_Neam%C8%9B_Citadel

Barbar means what it means in Greek, because that's where it's borrowed from.