Project Orion is a cocktail napkin estimate. The actual problem is much much harder.

But as you point out, anything that can travel between the stars can trivially wipe out all life on Earth without warning. They also know we exist already thanks to the changes in the chemical composition of our atmosphere. If they want to kill us then we're dead.