You make a very good point.
This is something that London (at least) does well. Council housing is available across the city and you will often find amongst some of the wealthiest parts of the city alternating blocks of council housing.
It might indeed be a pain point for those who bought their £1M town houses to then have to deal with the consequences of a whole council estate across the street but it has (so far) prevented the ghetto-ization of swathes of our capital which seems to affect Paris for instance which has less of a mix of housing.
your criticism is spot on. There will be a have / have not divide. It will make today's inequality look quaint. The top 20th percentile in IQ/productivity will live way better than the rest, but they will probably be creating 99% of the value.
So what's your solution? With the upcoming automation wave coming, UBI is way better than the status quo. What is better than UBI?
Clearly the answer is no, and it will result in the ghettoization of those that choose to take only UBI for income. This already happens with Section 8 housing, retirement havens (eg Florida), and so forth. There would be an even starker contrast between the haves (who perform "useful" work) and the have-nots (who subsist solely on UBI for whatever reason).