I think it should be fine if you form a very small group and only ever meet these same people (optimally outside with a bit of distance of course). It shouldn't be that much different from people living in a medium-size household and much better for mental health.