The system you are wishing for (i.e., live where the free market guides you) is already, in reality, the system we nearly exclusively live in.

The public effort to encourage mixed-income housing and mixed-income communities in order to fight the effects of concentrated poverty is not just about politics. There are real issues at play here, e.g., societal, economic, environmental, etc.

Also, consider the benefits, to yourself, of living in a mixed community instead of a socioeconomically homogenous cluster.