My mental model of bodyweight is that my body can adapt to use less energy to some degree, probably a few thousand kJ at most. If I exercise or drop my caloric intake and it's still within that range, I won't have to dig into fat reserves much. If I do more than that, I absolutely will.

As you say, doing low to moderate intensity exercise for a long time -- like walking 20km a day for a week, which is double what the Hadza walk -- is enough to cause me to lose weight unless I'm eating more to make up for it.

Yeah, everyone’s different. My intake is around 10k kcal when cycling throughout the day, and it’s like nothing. How much I climb matters though, by a lot (at least 2 000 m). Strangely enough, hiking doesn’t cut it for me, it doesn’t go down much even after 50 km/day, which is wild (I still do it though).