History is replete with fallen civilizations, lost civilizations, and civilizations we barely know exist. There are likely many who rose and fell and who we are wholly unaware of.

So no, growth is not inevitable, especially if you turn the livable atmosphere unlivable.