The counter argument goes something like this. Let's double every 1,000 years for the next 1,000,000 years that sounds easy.
Let's see that's only 7billion * 2^1,000 or err ~7 * 10 ^ 310 people. Hmm, there are only ~10^83 atoms in the observable universe. I guess they must be really tiny people.
So, yea it's not a problem in right now and probably not for a long time yet. But, as soon as you start doubling something has to give.
The worst is when population goes past carrying capacity and then crashes. That's when you get mass die-offs.
Population can settle down at carrying capacity simply through reduced birth rates matching up with death rates. That's the situation we see in much of the western world, with better availability of birth control etc.
With humanity, "carrying capacity" is unlikely to be food-limited.
The top of the curve removes that as we can't pay for the inefficiency's of livestock or the extra daily calories to be overweight and feed 50+ billion people.
Human population is expected to level off at 10-13 billion based on factors relating to social organization, not the 50+ billion that could theoretically be fed.
We've got a lot of efficiency left in terms of agriculture as well, so there isn't much reason to think reaching the carrying capacity of the planet is a particularly pressing issue. The amount of food that we already grow that we don't eat roughly 1/3 already, and the vast majority of land doesn't contribute to agriculture.