We have found Mars fragments on Earth. Some almost almost certainly went the other direction.

Fragments don't just lift out of a fully formed planet without a collision with another massive body. I assume the Martian fragments you mention are from the early solar system when things were still being formed. Also if something like the Chicxulub impact was enough to lift fragments out of Earth, I assume such fragments would have gotten so hot that any microbes on it just died, to say nothing of the trip in the void of space to Mars which would likely take millions of years before finally crossing paths with it. Good luck surviving the solar radiation and lack of atmosphere in the interim...

But it only needs to happen once, with one bacteria surviving the trip over the billions of years when both planets were hospitable to life.

I'd be hesitant to bet against those odds. I'm also not betting for it. Just being open to the possibility.