I don't consider myself an idiot but there are many times I read about actions taken by companies and go "S&*t that's legal!?!?!?". Many times unscrupulous and gray areas in the law aren't technically illegal. In addition the the Milgram experiments shows that even very smart people feel enormous pressure when when confronted with an authority figure. These two effects and I can imagine how hard it would be when the V.P. comes in and has a talk with you about how he had talked with legal counsel and they had looked over the regulatory rules and decided this was allowed, and other automotive companies were doing it.
The engineer could easily rationalize this away thinking "surely VW wouldn't do something illegal. They could get in so much trouble. I imagine I just misunderstood the laws."
The best explanation so far of how this may have come to pass is that the compliance department and the engineers that actually wrote the code report to the same person but not the CEO. That would allow the CEO to claim he and the board didn't know anything about this. That still leaves you to wonder how they could have been duped so easily, after all 'tremendous problem solved by magic' normally does not pass the smell test.
Now you're calling them dumb. That's probably even worse than malicious. This is one of the reasons why I am bothered with the VW explanation they actively picture themselves as incompetent because it looks better on your resume than 'criminal' and they are not at all worried about throwing VW as a brand under the bus.
There can be a legitimate reason to develop test cycle detection. And obviously, any ECU has the capability to put the engine in different regimes. Combining these two specifically for the purpose of cheating on emissions tests is not some major engineering project, if the pieces exist. In fact, you can make a decent argument that if you wanted to be maximally sneaky and evil, that's exactly how you'd do it. You'd keep the number of people without plausible deniability very, very low.
An ECU is not a 'big complicated thing', there is a ton of software in a car but the ECU itself happens to be one of the more easily understood parts where anything that does not have an obvious and direct function stands out like a sore tooth. There is also a very distinct point in time where the problems magically disappeared and of course it would be a simple thing to review the changes committed around and prior to that time. (Let's at least assume they have some kind of source code control system.)
If anything the auditors are more trusted than the programmers and they would definitely have had insight into the ECU code.
It is not conceivable that there is some kind of 'the cube' like conspiracy where the parts each individually look clean because one fairly large chunk of code was contributed by the Bosch company and contained a fairly explicit piece of code to detect the test cycle. If it's labeled 'process bananas' and an auditor is not able to figure out what it does it is their duty to actually dig until they understand that piece, that is what they are paid for.
So I believe that you are principally right that besides one exec and a bunch of people down from there in the org chart they kept the number of people as limited as they could but there is no way the compliance department was this incompetent unless a rookie was specifically tasked with auditing that particular engine to make sure the cheating stayed hidden. A seasoned auditor would spot this in a heartbeat and no rookie should have been able to sign off on these changes on their own.
And 'plausible deniability' is a very very thin fig leaf away from being culpable, all it takes is a single email or confession to pierce that veil and with the number of people VW is throwing under the bus (and at ever higher levels) it is a matter of time before someone decides to come clean about the whole thing.
To me, VW's story is at least somewhat plausible because it is not difficult to intentionally and specifically set it up to be that way. I don't have any trouble believing that they deliberately went to great lengths to ensure that and that outside of a small group of crooks (culturally and otherwise encouraged and enabled from above), it was borderline enough that nobody stood up and said 'this is illegal and effed up, I'm going home'.
As an aside, this same argument could also be used as justification for NSA programs that violate civil rights.