If they found a way to use more than one username they may very well have run it through the review process and approved it themselves
Programmers: read-write to repository
Staging team: read-only on repository, read-write to test servers and staging zone
Deployment team: read-only on repository and staging zone, read-write to production
It wouldn't prevent malicious code going out but at least would require a chain of cooperation between employees, which would be harder to achieve.
No, it just requires a chain of cooperation between authorized accounts. That's a very important distinction, especially here, where the email in question alleged the following:
This included making direct code changes to the Tesla Manufacturing Operating System under false usernames
* Storage engineers: Generally have access to most storage (all of dev/test/prod) in their group. Sometimes their access is silo'd, sometimes not.
* Backup engineers: Generally have read/write access to _everything_, and all historical versions of it, as backup systems need to be able to do both read/write. Fairly often there are ways for this access to be "unlogged" too, so the actions aren't captured into any system auditing logs (otherwise it can screw things up). I've not (yet) seen access for backup engineers ever be silo-d, but some places might be doing it and I've just never seen it. :)
It very much sounds like thats the case here - production code was edited, and subsequent auditing has found what should've been deployed and what is deployed differs.
And what stops a single member of the staging or deployment team patching the build scripts, binaries or just installing their own software to a server?
Given this, I think it’s much more likely there were few or bad controls, than a person in an incredibly privileged position working in the margins.
That's completely unnecessary and should not be the case. If you need something pushed quickly, you can get a colleague with review bit and get them to ack for "urgency" reasons after a quick lookover.
I believe this policy, used only a few times per year, has saved us 8 figures in outage costs over a decade. (More than half of the benefit is from a clear statement and instilled sense of ownership, and only secondarily the defusing/unraveling of people would otherwise wait or insert tangles of “best practice” red tape while the website or a factory was hard down.)
I based it on 14 CFR 91.3 (in intent) which says, in part:
91.3 Responsibility and authority of the pilot in command.
(a) The pilot in command of an aircraft is directly responsible for, and is the final authority as to, the operation of that aircraft.
(b) In an in-flight emergency requiring immediate action, the pilot in command may deviate from any rule of this part to the extent required to meet that emergency.
(I made part c of the law, the reporting requirement, mandatory where it’s only on-demand in the aviation law.)
When we explain the policy to new employees, we often cite the aviation law directly, to help clearly communicate our intent.
Often it boils down to not taking sufficient measures against social engineering. In this case the claim is they used fake usernames - most places I've worked, successfully getting a fake account if you already work there would tend to "only" require a willingness to lie on a form or two ("fake" a contractor) and then request elevated privileges. Very few places I've done work requires sufficient checks or counter-signatures to require additional accomplices or make it harder than that. They do exist, but they're rare.
The state of security most places is quite depressing at times. Then again, most of the time it's enough.
Not that that's necessarily better... Manufacturing equipment's at about the same danger tier as cars.
There is really no reason for this to be the case. Certainly all code that actually runs on the car can be required to go through review and be verifiably built, even if server code standards are more lax.