I pretty much agree, though really, a programmer who cares enough can carry you well past 5 servers. A good SysAdmin can handle a ridiculous number of servers; I've worked places with north of 10K servers per SysAdmin. (of course, here the hardware monkey and the SysAdmin job was split; you need more hardware people than that, unless you are outsourcing your hardware to amazon or the like. )
Splitting the roles is also practical. At some point you may have a project deadline coming soon, when suddenly (for example) network fails - do you really want to send experienced programmers on a trip to the datacenter in that case?
Really, it's a question of how big you are and how much time does your infrastructure need...