I agree. I think the problem stems from how we determine the caloric content[0] of a food stuff. We add the known caloric content of its ingredients, which we determine by burning them. This is probably just not a good model for how the body uses food.

[0] https://www.scientificamerican.com/article/how-do-food-manuf...