100 calories per hour mean an additional 571 calories per day if you spread it across the whole 7 days.

When your body needs 2600 calories per day just to stay at the same weight it is very easy to lose weight by cutting them to 2000 calories. When your body only needs 2000 calories then you need to cut them down to 1400 calories for the same result which may be impossible from a psychological perspective.

One free apple a day over decades makes quite an impact.

You need to eat about 7,700 calories too much to gain a kilogram of weight (https://www.livestrong.com/article/304137-how-many-calories-...). So, eating 25 calories too much each day gains you that kilogram in a year. If you keep doing that between ages 10 and 40, you ‘gain’ 30 kilograms, and chances are you end up obese.

You can counteract that by burning about 25 more calories each day. To do that by walking, you need to walk about half a mile extra each day (https://www.livescience.com/8931-calories-walking-burn-depen...)

The difference in the amount of calories I burn on days when I go to work, and am mostly sedentary, and days when I'm at home puttering around constantly, is massive.

At least as tracked by my Fitbit it can be upwards of 2000 calories difference, and 10-15k steps.

It's unlikely that your TDEE is 4000 calories when you're at home puttering around. These fitness trackers are extremely unreliable at estimating caloric expenditure.

Whether it is accurate in absolute terms is less interesting than what it reports in differing trends. Every indicator that it reports is vastly higher on the days when I am not in the office, chained to my desk. Just being on my feet, walking back and forth all day, up and down and round and round, is a night and day difference.