I think the key part of that is working on the same thing.

I work in a large company, and team is spread out over multiple cities.

We actually will often to group video calls to have a similar thing to what you describe - being able to ask a quick question, run ideas by other, etc.

However, going to the office I'm just with people who are working on different things, often doing calls which I find distracting...