We're mostly killed by things that have never been in humans before because we haven't developed defenses. Sure, most of those things don't kill us, but you really only need the right one to come along and basically wipe us out. Part of the reasons we're ultimately able to develop a defense is because it is Earth life and we've seen all the tricks. Any alien microbe that is able to establish itself inside our bodies might not have to worry about any of our defenses.
I think you've really got the wrong model. Think of the introduction of non-native species to various ecosystems. A lot of the time, the non-native species either can't hack it, or they find an equilibrium. But some of the time you get kudzu or eurasian watermilfoil.
In any case, the point is largely moot. Any civilization we are likely to meet in the next thousands of years will necessarily have to visit us and if they can do that, they can just lob a couple space rocks at Earth and wipe us out from a distance.
We have examples of exactly this! It was very bad: https://en.wikipedia.org/wiki/Native_American_disease_and_ep...
A bacterium that hasn't seem humans before is pretty unlikely to be able to do anything to us. One that hasn't even been in mammals before is even less likely, etc.
Random bacteria and fungii that are introduced to humans don't kill us all off. Bacteria especially are everywhere, it stands to reason that over all of the exploring, investigating and just plain spreading that we do, we'd encounter some bacteria that are new to a population. If those had any great chance of killing us off, shouldn't at least local examples exist of that happening?
Instead I think what we see are the organisms that affect us are those that coevolve with either us or with related species. We're killed by the things we spend a lot of time with, not random out-of-context bacteria that's never seen a mammal before.
If that's true of us, seems hard to say why it wouldn't be true of aliens, no?