I don't know if it's common to other countries, but a big factor in the US is race and class. There was a ton of "white flight" in the US starting in the 60s with white middle class and above fleeing cities to the suburbs. As such, they specifically did not want their adjacent town to be annexed by the major city.

In plays out pretty comically in Austin TX in my opinion. You have these teeny affluent enclaves (search for "Sunset Valley", "Rollingwood" and "Westlake Hills" near/inside Austin) that anyone else would consider "part of Austin", but they don't vote in city elections or pay city taxes (but they certainly take advantage of all the benefits of being close to Austin...)