I've done two playthroughs. The 1st playthrough was pretty lucky and I had no issues... but the 2nd playthrough of mine was "deadlocked" behind those Drifters for a long time.
Its very punishing to fall behind the drifters: it takes a lot of effort and optimization to catch up to them and overtake them. But I assure you, its possible, even if they kill all of your probes.
I would imagine that a lot of players get unlucky, and then get "trapped" there. Its the first time you see your numbers drop, since the adversary actually kills off your means of production. Growing (and out-growing the adversary) takes a very different strategy than the rest of the game up until that point.
https://universalpaperclips.gamepedia.com/Universal_Papercli...
You want to be over 10 Probe Trust when you start thinking about putting anything into Speed or Exploration.
Also, take one out of Hazard Remediation to put into any of the three Production skills as needed.
Once you see the percentage of universe explored start ticking up on the screen, the game is over. With a high enough Self-Replication, your growth is closer to exponential than linear. Doesn't matter if you're slow and don't explore far, you will just occupy the entire universe.