I've found these technical discussions, pair-programming, mentoring, architectural consensus building, are all much easier to do "remotely" than typical management-style discussions. Our project's technical team size fits into the "two pizza" rule, although just barely probably, so it's not like we're super tiny.
Pair-programming is great online. Better than in-person in my experience. Simply share your screen, and I can put it up on my second monitor. We've successfully mentored numerous junior-ish devs during COVID, with heavy use of remote pair-programming.
Architecture type discussions main drawback is missing a whiteboard, but there are a plethora of online tools of varying quality. In practice we often use draw.io and show a bit more finished copies than pure whiteboarding, but we've used other more simpler solutions for whiteboarding here too.
Design discussions/architectural consensus take place fine online, we just haven't had any issues. Just like pair-programming or architecture, we might screenshare but mostly we talk. And we find we don't lose anything from this. Sure, you don't get body-language, but that isn't needed for discussing bearer tokens.
In summary, while I agree that technical discussion is necessary at all stages, I believe that technical discussions is far less negatively impacted by remote when compared to non-technical discussions (i.e. managerial).
Yet somehow the world of open source keeps putting out high quality software with teams distributed across the planet in different time zones with different native tongues.
For me, my boss has told us he wants 2 people on site every day per week, doesn’t care who. In the event a new server or other hardware is shipped to the building he wants a safe two-man lift to be possible. I think it’s fair, and he doesn’t care who goes in, so it ends up being the people that like being in the office the most that show up.
Junior engineers need mentoring, need to talk to other people to understand what's asked of them.
Mid-level engineers/ICs need to talk to other people to understand the architecture design, collaborate with other mid-level engineers, and push back or report back issues that arise during the work.
Seniors and up need it for design reviews and achieving consensus.
Critically, to advance in your career, you need to start participating in conversations with the next stages. All of this is easier face to face.