I think the average person doesn't realize just how little activity they do in a given day. The thermodynamics still apply, you just might be doing way too much sitting proportionally in a day and not nearly enough activity, and that has a far larger effect than any variance in metabolism. We evolved as persistent hunters who would walk down pray over miles a day, foragers who would range around the world chasing resources, now with work from home your range might be a few dozen feet within the confines of your home on an average given work day.