It may not work for the five people in the office, because it’s not the environment they enjoyed if the 95 other people are not there. This is particularly true if teams are split that way.

Nobody should be forced to anything, but that doesn’t mean that an arbitrary mix works.