It doesn't have to be a problem, however sharing fewer interests is what I meant by 'gradual drift'. Someone I share 0 interests with that also doesn't just enjoy my company I will eventually interact less and less with. If you keep multiple diverse friend groups, you're definitely less prone to this issue
We just have a select few common interests and enjoy each other's company.
I have a different circle of friends for most interests, there's not even any overlap between the ones I online game with or dirtbike with. Hell there's not even overlap between the dirtbike friends and street riding friends. I have interests that aren't shared with any of them.
Buying a house in the city is just adding another interest to the pile of interests that only apply randomly to a small fraction of my friends.