0.1% of 1 million calories is 1000 calories, which corresponds to about 0.3 pounds. If someone gains or loses 10 pounds in a year, that means they were around 35000 calories away from equilibrium throughout the year, which is more like 3-5%.

So a 5% calorie deficit or surplus would have a fairly small effect during a single year, but over a decade or a lifetime the effect is huge.