It really is that simple, as you said in your post. Your metabolism (rate of burning calories) is not constant. Your actual consumption of calories versus your actual burning of calories dictates weight gain or loss (barring degenerative diseases).

If you are 800 calories, and burnt 1100, you would lose weight slower than if you ate 900 and burnt 1300.