How do you manage when different people in the same area are in different meetings? That's where I see the problems occur - there'd be 4 people in my immediate vicinity having 4 completely separate conversations. That would never work.

That seems like a different problem - I'd assume you have a meeting room for each conversation. In your case we'd have headsets at desks - we do that too. The previous question was how meetings are handled (in a conf room). Regardless, chat+conf for everything even if it's in-person in the same room (audio off if local).