Back before agriculture, Hunter-gatherers had a very slow population growth with a timescale of thousands of years. By the time they started getting crowded, some startup discovered agriculture and bread and beer and now we could support a larger population and a higher growth rate.

Living without growth would be hard too. The zero-sumishness which characterizes our politics would be even worse.