> one of the people present said that they missed being able to just pop up, look over direct reports' shoulders, etc.

Sadly, I think this is a huge part of it. Regardless of productivity, "I just want to stand up over my empire and hear the bustle of people in my org talking and doing! Visibly see the movement and activity of My People physically surrounding me, doing Business Things!"