Any large project will have plenty of technical debt that isn't important to clean up now but should be done. There are always new tools to try and see if they add value to your project. There are new frameworks to try that might or might not be worth telling everyone to start using for the next story.
The above does work for a tiny company of course. However for larger companies it is important.
How do you retain your smartest/best devs when you are putting them on the least important projects and expecting them to tolerate infinite interruption?
Certainly there is a tradeoff of coaching vs doing for seniors, and you want to raise your overall team up.. but in practice a lot of "agile" environments are very focussed on output, and want to see more points/stories out of higher paid people. It sets up an adversarial system where nothing is expected of some, and everything is expected of others.
You are also correct about team size. In an engineering org with 1000s of people, you want everyone to grow and no 1 person really matters.
In startups, small orgs, and teams of 3-5 devs.. you do need your best doer to, actually, well.. do.
The smartest person on your team is going to be interrupted the most. The worst person on your team is going to do a lot of interrupting. So your highest quality producer will produce less & your lowest quality producer will have their incompetence papered over. With remote, a lot of this interruptions are now on slack, and searchable.
In an environment with a lot of juniors that need coaching, the office definitely excels. For a team of mature engineers trying to work on challenging tasks, it can be highly disruptive.
I've been in a fully remote job and gotten more built in the last year than in probably my previous 5 years in office. Meanwhile when I was in-office, most of my coding was after hours (back at home).
Remote also means I spend less time in conference rooms listening to monologues and more time on 1-1 or small group zooms with productive screen shares of code/data/probelms.