Anecdotally, people adapt to do whatever they are doing efficiently. At some point I started exercising (distance running) a lot and I lost weight despite eating constantly. That phase lasted a few months, at which point my hypothesis is I got efficient at running and no longer burned huge calories doing it. I started gaining weight and had to adjust my diet.

Seems to me the example of the hunter gatherers is the same. Their bodies have adapted to do their everyday tasks efficiently. This is a logical evolutionary adaptation. It also leaves room for exercise to play a role in weight loss, by doing things you're not well adapted for. Diet of course is also important, however both have a role.