Hence the line about people being pushed to the outskirts.
Now hunter-gathering is a subsistence lifestyle. Back before agriculture it was a very different story, since there would have been hunting in the most productive lands, too. Fossil records back this up - early hunter gatherers were much larger and taller than later farmers.
It's true there are some climates/geographies that will support some small amount of human life year round with little effort.
There are many others where most of your day will be spent chopping wood, making shelter, and preserving enough food to make it through the winter.