I realize that they probably have seen some bad things happen from contacts with the outside but I doubt they have an understanding of germ theory of disease yet. I assume they have stories they repeat to one another, maybe songs or warnings they provide to one another to encourage this ruthless xenophobia. I'd love to know what those things are.

It's not hard to make a connection between "a foreigner shows up" and "people suddenly start dying of weird unknown diseases" when the entire population is small enough for everyone to individually recognize everyone else.

Do they actually know that though?

Yes? You don't need science or even literacy to tell your (grand)children that everyone who meets outsiders die messily (cf. All Andamanese except for 80-strong Jorwa(?) and this North Sentilenese tribe)