We had the Red team come in and while pentesting share his screen with us all. Another Red team member explained what he was doing and after an attack was launched and we would see if our tools detected the activity. If they didn't, we went out to find out why. This was huge. It showed us where we needed to tune some things and where we needed newer and/or different tools.
This isn't the only way we get pen tested. They do their annual "regular" pentest. The Purple team thing was awesome though. We learned a ton. Since I happen to own most of our tools and am secondary on the ones I don't own, I have learned a tremendous amount and I've been in IT for 20 years.
I think there needs to be greater punishment for companies that lose customer information. Only then will the incentives be large enough for something to be done.
My experience is limited to security software development in e-banking, e-commerce, network security and data security domains in technology areas like cryptography, PKI, deep packet inspection, and network protocols. I know my experience may not be representative of the entire security industry and there is a possible selection bias too (i.e. I may have seen more demand for blue team engineers because I have belonged to blue teams myself), but I thought I should share my experience here to present the other side of the story.