Civilization also can mean high population density, far more intricate social networks and occupations, intensified warfare, possibility of better nutrition, more need for long-term planning to save up money and manage agriculture, etc. (Of course, civilization can also work the other direction as the past few centuries suggest. That there might be a long run trend to select on intelligence doesn't mean that there are no short-run trends select against it.)

Doesn't surviving and thriving w/o civilization take more effort mentally and physically? Isn't civilization a labor saving device, and viewed as such the tool user doesn't need to be as skilled or strong.

It has never been easier to kiss your bosses ass, take an early bus home and order a pizza with your smart phone en route.

Physical sure (so far as strength and endurance). Mental though, no way. Because in a populated area the competition isn't nature, it's other human beings, and outsmarting them is key.

It's important to remember that the metric of success in evolution is not outsmarting other people. It's how much you can spread your genes!