Nerds dramatically exaggerate the significance of this. Yes, if you just look at the kanji, "outsider" is possibly the more literal translation, but at the end of the day what matters is what is in people's heads when they use the word, and not the word itself.

"Gaijin" gets played up in weeby debates -- by people who are looking for reasons to be offended -- as some hugely significant cultural deviation, but it's honestly more like "foreigner" in practical usage. That word can be incredibly rude, or completely innocuous in different contexts. As can 外人. It's more accurate to say that both words mean "foreigner", but 外国人 is the PC version that renders you safe from criticism.

(The irony, of course, is that no matter where you go you'll hear plenty of borderline racist policy talk with the sweetest of language.)