There's a lot more to society than energy. Take food, for example - a huge portion of the world's food is grown in the American midwest. That food is powered by fertilizer (petroleum) and aquifers that are being depleted far faster than they're being refilled. Or seafood, which is being depleted rapidly. More people require more resources - wood, minerals, you name it - and there simply isn't enough to go around arbitrarily.