Your 'one thing' is exactly what Thomas Malthus originally wrote about in the first well-known book on population growth!

His thesis, glibly summarized, was that population growth gives the rich a structural problem: given that reproduction is inherently exponential and the poor reproduce more than the rich, the only logical consequence is that poor people will overwhelm the rich.