You would be surprised. Most companies I work with are operating on only 1-3 machines, and using the CloudFront infrastructure to distribute content to edge locations around the world. They a handful of Lambda functions to trigger various random tasks like backups, notifications, git pulls, deploys, etc. Oh and S3... tons of S3 storage.

I have a client that does $50M a year on a SAAS product using only two EC2 isntances, but 1 of those instances is a developer/beta/test server. Only 1 is used to manage sales across the entire world. They rely heavily on Lambda and SNS & SQS as well.

Another client operates eCommerce at around $80M a year revenue using 3 EC2 instances. One of those is a test/dev server and the other 2 are used to distribute load and for redundancy with a load balancer sitting on top.

I have 2 other companies that are smaller that generates about $2M and $5M a year respectively that both run a static site hosted on S3, using Lambda for deploys and cloudfront for distribution like mentioned above.

Point being, you would be surprised how low key many company's AWS deployments are.

---

If I could pin one skill that will stand you head and shoulders above the rest among cloud architects, it is to learn Serverless functions. In this AWS world this is "Lambda". Microsoft calls it "Azure Functions" and Google Cloud Platform calls it "Cloud Functions". If you can write serverless functions AND are AWS certified, that seems to be the hottest thing that I have seen.

That doesn't really surprise me. It just strikes me that the level of knowledge/skill necessary to manage that isn't all that great.

It especially confuses me that people want people to "learn" serverless functions. Serverless functions are just like normal functions, except with a library to use for communicating with the outside world!

If I was hiring, then I'd be asking people about VPCs, IAM roles and Security Groups. Those seem to be the complicated bits that you need to understand to do pretty much anything. The service-specific APIs seem relatively simple...

Let’s be real - how many business projects honestly require tens to thousands of “machines”? And what fraction of IT people work on those projects?

There is a huge amount of IT work that never gets to that scale, and of the IT projects that get there, lots don’t actually need to be.

I guess it's all part of the "silver bullet" and "hype" culture. People want something that will automatically scale their new giga-scale web product that is going to be "the new facebook!", and managers want something that will make them rich without too much work, so they sign up for all the latest cool tools instead of thinking deeply about their real requirements and good old fashioned hard work. Some of the latest cool tools really _are_ amazing and worth every penny and more, but I'm sceptical about some of them...

> The tricky part seems like it would be dealing with tens-to-hundreds-to-thousands of machines which should automatically deploy and scale

You must be dealing with exceptional companies who have application which need auto scaling. And need alone is not a sufficient condition, application should support it too.

Most application are simply lifted from legacy with no code changes and serves at-most a few hundred users at max and will crash and burn if autoscaled.

Autoscaling is not just about scaling. It’s also about high availability.

I have two or three ASGs up with a minimum and maximum of two and an HTTP health check that will kill an instance if the health check doesn’t pass and bring up another one.

I also have applications that autoscale based on queue sizes.

Auto scaling means you will be provisioning the server on the fly when needed which manes names, IP addresses are not important.

Scenario I am referring to involves application team closely guarding their servers like babies which means they are married to server names and IP addresses. These are cloned from on-premises and load balancing is just sticky sessions going to one node with other one sitting idle until first one fails.

And that’s a bad design. The limit of the amount of times that I will get up in the middle of the night because of a non redundant, non HA process is one before I put that on the top of the list to redesign the architecture (not the program necessarily).

I know but that's how risk averse people at large companies behave. They are a) they lack understand and b) risk averse because risk reward ratio is high.

First group is rare.

I think auto-scaling is only really relevant to 12-factor-style stateless applications. Otherwise you have to do sticky sessions, and that complicates things considerably, and isn't even a silver bullet.

Even with on prem solutions, I would hope that most websites are behind a load balancer with at least two instances. Storing session state on the web server hasn’t been something that I’ve ever done and I have been developing websites since at least 2002.

Legacy applications may not be websites, and might be stateful (or so I am told).

But how many store data locally as opposed to in a database? Even for stateful applications, if you have the source code, you could put an http listener on it for your target group to ping as a health check and put your data on FSx. If the app became unhealthy, kill the server and bring a new instance up. Have a minimum and maximum of one.

I wouldn’t go that far though. I would probably use something like Consul to do a local health check first and restart the app if it dies and then do the target group/health check as a second level check.

I was thinking more like some off-the-shelf solution purchased years ago that is something of a black box to those that operate it... subtle failure modes, lack of instrumentation, and opaque inner state.

Agreed but IT bosses just picks up these keywords and start throwing them around as benefits of cloud. Heck I had a CIO getting excited about moving to micro services when his entire landscape had only COTS application.