This is very common among endurance athletes. It's extremely difficult to consistently recover from a 1,000-2,000 calorie deficit every time you do a weekly long run or ride. Glycogen reabsorption is optimally effective for only a relatively brief time period post-exercise and there are upper limits on your rate of glycogen resynthesis.