Some of it is driven by economic growth (city centres have become increasingly unaffordable, particularly in capital cities), some by population growth (you can only scale up a residential block so much before it becomes a mess from a social perspective), some by fashion (hello Hollywood), some by speculation (it's easy to bootstrap a shopless residential suburb), some by urban planners simply failing in the attempts to create new urban centres.
Building in a conservation area (not necessarily rural!), on a listed building or in a "greenbelt" area require special permission, which is not always easy or obvious.
I think the theory must be that people want some assurances. If somewhere is zoned for housing, I don't have to do pointless paperwork to build houses. It also keeps commercial areas away from residential which might or might not be a good idea although I suspect most residents prefer it that way.