Not necessarily that colourful people leave, but that outsiders leave. For one because they can and have somewhere else to go to, but the other because if they stay they will be viewed with suspicion and bad things might happen to them.

The Witcher series has been full of fearful populaces wary and hostile to outsiders/other races (Dwarves and Elves in this case).