The rest of what you say I think I agree with (though I have no idea what a low albedo impactor is).
Note that they only have to be friendly to themselves to survive to get into space. What if the mechanisms that ensure their peaceful cooperation just don't apply to naked apes? Just one example: we say that a smile is "universal" but that's just because it's wired into our species. An alien species would've been subject to an entirely different evolutionary context. A smile would be meaningless to them. We still don't know everything about how altruism works in our own species. I posit that we have not a clue about how it would work with aliens, or even if it would exist at all towards us.
(Another example: when was the last time you felt pity or forbearance in response to a reptilian submission signal?)
I have no idea what a low albedo impactor is
It's a kinetic projectile that doesn't reflect a lot of light.
But plenty of logistics and game theoretic ones.
Logistics: An expansive starfaring culture can garrison every rock in the galaxy within a few million years. The fact that they haven't means that either (1) no such cultures exist or will exist in our vulnerable window, in which case we are unconditionally safe, (2) the galaxy is already garrisoned and they are tolerant, in which case we are conditionally safe as long as we don't piss them off, or (3) we currently live in the special 0.1% of the galaxy's life when a hostile culture is expanding but has not reached us yet, in which case we survive mostly by random luck. The likelihood is that we are in scenario #1 or #2, but cannot know which.
Game theory: A group is deterred by internal risk analysis, not external threat. A clever expansionary culture will run the logistics analysis above and decide that hostility has a horrific risk of backfiring. (For a fictional example, see Greg Bear's Anvil of Stars.)
If the aliens are still living in cities and orbital structures, then they'll want to wipe out a star faring human race. Any species capable of star travel can also accelerate and target enough low albedo impactors (perhaps relativistic) to wipe out a planetary or even system-wide civilization. There would be no warning, so there would be no possible deterrent. The only logical way to avoid genocide would be to strike first.
If the aliens have uploaded their consciousnesses and have all become spacecraft themselves, then we would be safer when encountering them. Wiping out such a civilization would be too difficult, so there would be a deterrent factor working on their behalf. There would be no need for them to act first to avoid genocide, and there would be no options in our case.