I think some of us genetically are more sensitive to the impact of light, caffeine, etc on the circadian rhythm. There's also other factors like work. Cortisol levels are higher coming off of work and it takes time to wind down.

Think of it this way: there's nothing special about a time zone, and yet we know that if we moved several time zones away, we'd shift. The differences are in light patterns, timing of work and social life, food intake, etc. Incidentally, many of these factors are within our control.