Sperm count has to fall quite a bit further before it becomes a bigger limiting factor than women actively chosing to not make babies.

Also, I'm not making predictions about "forever". Rather, my prediction is that the reproduction rate is likely to come back above the replacement rate very quickly, compared to evolutionary time.

My guess is somewhere between 100 to 1000 years, if human societies go on approximately like today, and we don't have some kind of apocalypse, AI takeover or totalitarian world government.