True, but that a form of reasoning inevitably gives a wrong conclusion under rare circumstances is only a weak argument against it.
-Based on our experience, it takes ~4bn years for life to evolve in a planetary system.
I don't think we have a strong reason to believe that there's something special about that length of time. Not much change happened on Earth for most of that time, suggesting dependence either on stochastic evolution of a few things like photosynthesis (so 4bn probably isn't much more likely than 3bn) or on something like increasing solar output crossing a threshold (it's doubtful such a threshold would be crossed at ≥ 4bn years at every location). That aside, even a few tens of millions of years is plenty of time for life to cover a galaxy.
You could make a good case that we are part of this first wave: -The stars created after the big bang would not have had any heavy elements and therefore wouldn't have had the resources to create life. -These stars lasted ~10Bn years before they started to go nova (which creates the heavy elements for the next generation of stars) -Based on our experience, it takes ~4bn years for life to evolve in a planetary system. -Therefore the first lifeforms wouldn't appear until roughly 14bn years after the big bang. -The universe is currently 14bn years old.