I'm confused about what you're arguing. Are you saying "by allowing people the freedom to work from home it harms people who wish those other people would join them in the office"? Or are you saying, "I realize you're not saying people should be forced to come into the office, but if you were arguing that it would be very selfish"? One of those is absurd and the other is the weirdest kind of straw man I've ever heard.