It doesn't have to be a problem, however sharing fewer interests is what I meant by 'gradual drift'. Someone I share 0 interests with that also doesn't just enjoy my company I will eventually interact less and less with. If you keep multiple diverse friend groups, you're definitely less prone to this issue