Completely agree.
> Want ssh access to production? No problem.
This makes me a little uneasy, I'm not sure everyone should have ssh access to the production server.
Every _developer_ should. No question about it. Sudo should given on an is-needed basis, but ultimately, as a developer I can screw up a lot more by simply writing bad code.
Simple philosophy: you build the software, you should be involved in running it (including carrying a pager). Amazon's CTO agrees: http://twitter.com/#!/Werner/status/50957550908223490
Fair enough.
This, more nuanced point, I agree with.
Ironically, if you treat production as an alien land developers aren't allowed into (and have no transparency about), you're going to create an environment where developers completely ignore operational concerns like security (e.g., having no authentication mechanism on their own services, as it's presumed production is a "magically secured" environments where no one may connect to in any way).
For mitigating exposure to crackers, though, it makes sense to minimize the number of possible entry points someone could compromise in order to put malicious code on your production servers. The source control system (did they really not have a source control system!?) is a less vulnerable avenue than ssh, because presumably third-parties review what flows through source control before putting it on the server.
Ssh access doesn't have to come with privileges: main purpose of ssh access is to be able to run top, iostat, ps, strace/dtrace, grep log files and also verify that my service is configured correctly.
You are correct that code can be reviewed, but that isn't always the case nor is the reviewer omnipotent. In any case, with both code and ssh is there is a strong audit trail: an employer needs to make it clear which are fire-able offenses and which aren't.
For what it's worth, "give developers read-only ssh access to machines that don't contain sensitive customer data" works great for Google, Amazon (where it also comes with a pager, something I'm in favour of), LinkedIn (recently implemented-- this made my work much easier), parts of Yahoo and I'd be surprised if that isn't the case at Facebook. In other words, companies that are strongly oriented around UNIX/Linux (it's available as an option on developer desktops), which can afford to hire (and are able attract) strong developers and strong operations engineers and which are in the business of writing Internet applications.
My personal philosophy actually goes quite a bit beyond that: hire great, generalist engineers who are considerate and nice, give them root. Let them push some code without review, if they're confident their code won't cause damage. Review any tricky code, bug fixes, or mission critical components (e.g., the HA storage system, revenue loop components, UI changes). Roll back instantly if it trouble occurs (something you couldn't do at MySpace, apparently!).
If someone cracks your developer's development workstation, they can piggyback on that developer's access in order to insert malicious code into a commit, or in order to ssh into a production server and run a canned exploit of a local-root vulnerability. The first of the two leaves a strong audit trail, and may require a third party to sign off on it before going to production. The second probably doesn't, and won't.
If you can run strace on a process, you can inject malicious code into it.
While this is a theoretical consideration, I don't know of any security breaches due to this policy at the companies you list. On the other hand, there were security breaches at MySpace due to gross incompetence on the part of the developers — most of all, Samy is my hero!
I wasn't suggesting that developers themselves would be putting malicious code into production.
You are also forgetting that there is usually a step between a developer workstation and production, and at that gateway you'll typically have additional security measures (so that simply getting to the gateway doesn't get you to production).
I don't, however, disagree with your overall idea: yes, technically, developers having ssh access to production might (to a very small degree) reduce security, all else being equal. However, there countless benefits to giving developers ssh access that result in greater security.
Nor do you have to use the same policy for all machines: SOX, for example, mandates that developers that write the code that handles financial transactions shouldn't have access to machines that run this code (to prevent fraud). There are other types of machines I'd include in this case (databases holding sensitive user data, machines holding sensitive configuration, etc...). However, for a vanilla machine running an application server, or a database server holding strictly non-sensitive/non-revenue data, that's not the case.
There are also far worse mistakes one can make (e.g., don't use version control, don't put proper review procedures in place, hire/don't fire incompetent developers) which will impact security.
The comments state that there was no staging or test environment, no ability to roll back releases, refactoring was a dirty word (engineers wanted to refactor, but couldn't), principal on technical debt was never paid (only the interest in terms of hacks to make the site scale-- again, product organization prioritized new features).
The rest: location, technology choice aren't sufficient to "kill" a company: there are successful companies in LA, there are successful and agile companies using Microsoft stack (where appropriate-- see Loopt and StackOverflow/FogCreek as examples of companies using both FOSS and .NET). On the other hand, they're not optimal either: they aren't what engineers would choose themselves most of the time.
This indicates that the technology and location choice aren't the cause, they're symptoms of company management that doesn't understand building non-trivial Internet applications (what the optimal technology stack for one is; where, when and how to hire developers to build one) and yet maintains authority to make all technical decisions. Contrast this with Facebook, Google et al-- where every process (from corporate IT to production operations to HR and recruiting) is designed with needs of the engineering organization in mind: "Want two 24" monitors? No problem. Want Linux on your laptop? No problem. Want ssh access to production? No problem. Want to fly in a candidate to interview? No problem."
[1] I personally wouldn't touch Windows with a sixty foot pole, but speaking objectively C# >3.0 and (especially) F# are great languages.
[2] "They weren't talented": having interacted with some ex-MySpace engineers, this just isn't -- at least universally -- true. Indeed, here's a secret to hiring in an early (Facebook in 2005) startup: seek out great developers who work for crappy companies, people who have joined "safe bet, resume brand" companies like (post ~2003) Google aren't likely to join you until you've already become a "safe bet, resume brand" company (Facebook in 2008-Present).