If you removed that perk those high functioning team members may not be able to produce at the same level at the office. Those leadership points may not go as far because you have to deal with a host of other issues. There are less sexual assaults and unwanted touching in purely remote teams. Less conflicts with culture, race and gender. Issues like Jim's lunch smells and is making Jan sick or Bob talk too loudly to clients on the phone beside Jim can he stopped.

Working from home can make a manager feel less control. It takes a different type of leadership. But requires less leadership points.