There's a great New Yorker article [1] which addresses this -- particularly that "pretty clear that this was not right" might not be that clear after all. Excerpt:

"...sociologist Diane Vaughan described a phenomenon inside engineering organizations that she called the “normalization of deviance.” In such cultures, she argued, there can be a tendency to slowly and progressively create rationales that justify ever-riskier behaviors... If the same pattern proves to have played out at Volkswagen, then the scandal may well have begun with a few lines of engine-tuning software. Perhaps it started with tweaks that optimized some aspect of diesel performance and then evolved over time: detect this, change that, optimize something else. At every step, the software changes might have seemed to be a slight “improvement” on what came before, but at no one step would it necessarily have felt like a vast, emissions-fixing conspiracy by Volkswagen engineers, or been identified by Volkswagen executives. Instead, it would have slowly and insidiously led to the development of the defeat device and its inclusion in cars that were sold to consumers."

"But assuming all of this is true, how could it have persisted for so long, and without more people stepping forward and speaking out?... Volkswagen was heavily invested, both financially and culturally, in producing a clean-diesel engine. That the company was failing to meet the standard required by American emissions tests would have been embarrassing and frustrating to its German engineers. Some may have seen those tests as arbitrary, and felt justified in “tuning” the engine software to perform differently during them—even as it now looks, to the outside world, like an obvious scandal."

[1] http://www.newyorker.com/business/currency/an-engineering-th...