There may never actually be hard data on this, because it's, at core, likely very hard to measure, but I'd love to read some research/findings on this issue more.
I'm in a coworking situation now, with a private space, and I'm always more productive in the private space. But I do relieve some stress and get energized by interacting with others in the common area. And in previous jobs, there's of course always times when you need to talk to colleagues about project issues, and that's a requirement.
To the folks (devs) who insist that they are "more productive" in open plan offices, I wonder if the rest of the folks in the room with you are equally as productive, or if the productivity is just being shifted from some people in the room to others.
The other is where you're solving a problem that isn't in the first order a code problem. There are typically humans involved in these kinds of problems. Most instances where the problem is defined as "building the right thing for the customer" where part of the problem is coming up with "the right thing". This works well in open plan offices where multiple people can quickly give input/get feedback. Remote working is probably not a great idea. Most work in early startups is probably in this category, and that also offers a model to explain why open plan offices are so common in not-startups-anymore: these companies are very eager to retain the energy of their early days, and a lot of that energy is tied to the collaboration of the open plan office, even if not necessarily suited to the workloads of the more mature company.
Tl;dr: when people speak of being more or less productive under certain conditions, a lot probably hinges on what kind of work they are doing.
Investing in people at the end of they day has to make sense, otherwise other companies will out compete you. When it's hard to determine people skills, work varies highly in intensity, projects fail for any number of reasons, people don't stay very long at even the most "comfortable" companies etc. the personal productivity of any employee becomes marginal, at least in the lower ranks. This is true for most "craft" industries like newspapers, fashion or design.
Well except one place where the people in the office where just horrible (really toxic culture largly down to the 'office manager' been about one step short of invading Poland) so I just did as I pleased (wandering in at 10am wearing flip flops and combat shorts), I was pulling 60hr weeks so the boss didn't care and she hated me anyway because she had to do payroll and knew what I was earning, I was knackered from day one on that job.
What I don't hear is the root cause of the angst. Is the sales team overpromising? I don't know. But when something's not ready for their demo it gets quite loud.
But with a huge caveat:
I can largely turn off other sensory input. Some is nice: a breeze through the window, a buzz of others around, nice light and open plan but not vast open plan; no one is no more than 4 desks away from a window (a proper window that can be opened and that has some kind of view). And cubicles shouldn't have huge walls; can see the face or top of the head of the person next to you.
Also: Lots of small meeting rooms dotted around with almost assured availability of one at little notice. It could be for a sensitive call, impromptu conversation, or just to sit in the dark working something out.
I couldn't survive normally in one, nor in the other. However I default at open plan. If others want to default the other way, that's good too. There should be options.
In my experience the only people who like open office are those whose job is to talk all day long. Sales guys, visionaires, busy bodies etc. Everyone else just suffers.