> You could manage a thousand servers, but ONE PERSON just can't operate thousands of servers.
Yes, I literally managed AND operated ~5500 linux servers in a facility. Remote power management, serial console access, PXE booting. I would replace any failed hardware when convenient, usually after my morning coffee or in the evening if I wasn't in a rush to get home.
> also how did you managed file access in a scaling manner?
Shared NFS with work distributed to servers on-demand or scheduled, based on workload.
> and when yes how did you provided your developers that space?
Limited SSH access to their environment; diskspace was provisioned in minutes. Scheduled jobs could be executed using an open source scheduling framework.
> did you created some kind of infrastructure for that (libraries, etc)?
No, the majority of the tooling was a few thousand lines of bash scripting
> There are so many things, which just doesn't work out. As already said, you've just managed it. But other people did the work, that this could work out and they needed a lot of time to get things done. Take a look at google, they always try to bring costs down, through things like borg, kubernetes, etc. but they throw thousands of people at those problems, also at the networking thing, they have a huge team of PROGRAMMERS that try everything that the management of their network will be easier. also if you don't need things all the time, why should you buy it?
Not knowing how to do something does not make it impossible or hard, simply an unknown how to do it.
PS: I could see a team of 10 people handling 55,000 servers though. But that's serious scale and fairly lean.
To directly answer your question, people from other teams were available if for some reason I was not.
Facebook is doing 1 engineer to 20K servers: http://www.datacenterknowledge.com/archives/2013/11/20/faceb...
The number of companies who can do that are single digits, but 1 person per ~5000 servers is absolutely reasonable.
I would also be suspicious of 1:10,000+ numbers as they often don't include people that are part of the server team but not standing in the data center. Aka Facebooks 3 person dev team, a manager, whoever designs there HW, the team spun up to do mass HW installations etc.
In the end AWS covers a lot of seperate costs which are only really obvious as you keep scaling up.
I'd like to see a similar analysis for say 50-100 servers, and see if AWS really costs more vs. colocation when you add in the resources to setup and maintain the infrastructure.
also how did you managed file access in a scaling manner? did you've setup hdfs? and when yes how did you provided your developers that space? did you created some kind of infrastructure for that (libraries, etc)? There are so many things, which just doesn't work out. As already said, you've just managed it. But other people did the work, that this could work out and they needed a lot of time to get things done. Take a look at google, they always try to bring costs down, through things like borg, kubernetes, etc. but they throw thousands of people at those problems, also at the networking thing, they have a huge team of PROGRAMMERS that try everything that the management of their network will be easier. also if you don't need things all the time, why should you buy it?