To create good solutions that are also innovative, I think that programmers only really need enough knowledge in that domain to be dangerous - to know more or less what happens but with enough gaps to not really know what's doable and what isn't. But, this knowledge is only useful if you can combine it with bits and pieces from several completely separate domains.
If innovation is the goal then I think there is a point where domain knowledge actually hurts you.
Of course there still needs to be at least one specialist to keep everything on track though - and I completely agree with you about having someone with deep knowledge on board. I think it should be someone who has been in that industry and will stay in that industry.
I can't see the point in programmers trying to become experts in areas that they're not interested in working in for the long haul. It's easier to learn (and actually apply) a little about a lot than a lot about a little.
One thing that I have always had issues with when gathering requirements around something for an unfamiliar industry is that people who are in that industry will often neglect to mention things that are both important and non-obvious/counter-intuitive to someone unfamiliar with the industry.
This is probably because they are so familiar with the quirks of the way that the particular industry works that such things seem so obvious as to not need mentioning (and they probably don't see how it could possibly be any other way). Basically similar to how we are often amazed by assumptions some people have about the way that computers/software works.
The paradox of this though is that to deliver truly good software for a particular industry one must really know that industry as well as somebody who has years of experience in that area. At which point you start to feel that you could do their job too.
Another alternative is to have a very tight feedback loop with someone who has tons of domain knowledge and is the target user. Basically sit next to them as you develop the software and have them review every single change you make.