The other thing is that because we have a fairly... ?linear record of evolution: if life did originate somewhere else it got to Earth in something like prokaryotic single cell form 3.5 billion years ago. Which means we can't use the idea of panspermia to tell us anything. It's neither predictive or explanatory. It tells us nothing of how or where life did originate or how life got started on Earth or what or where we should look for or expect in the rest of the Universe.
The current theory suggests life could pop up extremely quickly, which itself already implies life is likely to exist in the rest of universe on some level so panspermia doesn't even mean anything with regards to how likely life is anywhere else.
Clearly if you see humans living all over the globe they almost certainly originated elsewhere. But if humans only live on LI, then it's reasonable to conclude they originated on LI.
I feel like people get sidetracked into talking about evidence, when the issue is that life originating on Earth should not be the default assumption. I feel like someone more lucid than me could make a good Bayesian sort of argument.