Part of "that other stuff" is that the body do not burn or metabolize consistently when you change diet. For example, the body can go into a starving cycle which increase metabolism, lower energy usage, and make you less sated, and which reportedly takes a rather long time to recover from. It should not be that surprising that the human body has evolved to adapt when food availability change.