> Don't do that. No one is allowed to ssh to boxes. If you need to enforce it by blowing up and rebuilding all servers once per week, do that.

Yep, this is change control 101. I used to have a boss who would go in and edit sprocs on the production server and never get them into source control. I finally just encrypted everything on the server so the only way to push out new code was to commit to source control. A sledge hammer yes, but sometimes it is required.

Use Terraform or some other tool to codify your entire infrastructure.

>You should deploy so often that it is routine. Deploy 50 times a day. You will find bugs at first, but eventually you should get to the point that you could deploy for every single commit and no one would notice. (Now they may not be a good idea depending on your risk tolerance and other things, but you should be ABLE to deploy every single commit).

This is a great point. At first it is terrifying, but then you realize that a deployment is so easy that bugs are generally not a big deal. Bugs happen, so the goal should be to shorten the time between bug found and fix deployed.

> Don't do that. No one is allowed to ssh to boxes. If you need to enforce it by blowing up and rebuilding all servers once per week, do that.

I'd love to do that. I'd love to have no access to production servers, but ultimately that requires far more work to get right than Ansible configuring the same machines again and again. It also means you can't use dedicated hardware as easily, which restricts performance. It's a great situation to be in, but difficult to get to and requires a non-trivial amount of overhead.

At my place of work we deploy somewhere between 5 and 40 times on any given work day (on a team of 5 engineers). That's because we've managed to engineer a reliable and fast deployment process, but that took a long time to get right. It's powerful, but the overhead, particularly on a small team who are under pressure in a startup environment, can be quite large.

I'm not saying you're wrong, in terms of best practice I completely agree, but when the tradeoff is between sales/acquisition/product market fit/etc, and having a 'smooth' devops process, in many cases, the latter must come second.

You can devops dedicated hardware. It's a little bit different but not that much. Heck there are boot2docker and such that let you just run docker on bare metal.

>docker >bare-metal

To some, no virtualization is still best.

Edit: you can still PXE boot immutable images.

I'm not sure boot2docker even has a virtualization level. Havent dug super far into it.

This is so weird. I hear this all the time. At my place of employment, we all have the ssh keys into our EC2 instances, but no one configures them. Ever. Period. Those ssh keys are purely for either validating changes in a test environment (like to .ebextensions) or diagnosing production issues (why did Puma fall over this time? why isn't syslog output making it to loggly?).

Of course, we lean heavily on Elastic Beanstalk; autoscaling regularly kills old instances daily since we scale from 2 to 18 and back to 2 instances in a 24 hour period across about 9 microservices.

So, if this ssh into boxes and change things is common, it means people aren't doing auto-scaling? THAT is scary.

I'm very very pro devops. But I am cautious about auto scale.

It adds a lot of complexity. If your app has predprtable load it may simplify stuff a lot to not auto scale. Think a b2b app with manual account creation. You know your user levels. 0 reason to turn on auto scale.

...so you run the servers you need for peak load 24/7? Ew. So much money just thrown away because "it's too complicated". We just scale based off of Network Out and only really spent a week finding the right threshold.

How long does your stack take to add a server. 3 minutes?

I have created a lot of services that couldn't have crappy perf for 3 minutes every time the load scales. Especially since in many apps the most expensive part is storage and dbs which cannot auto scale.

So say you spend 1k a month in your dbs and $100 for peak web load. You could spend a week fixing auto scaling bugs and try to save $25 by scaling web nodes down... But then you still have bad perf every time load spikes. I would not do that to save $25. I would question a company I work for that did that.

You need to be scaling up and down for hundreds of dollars per swing and have big spike loads before you add the complexity of auto scale.

We're sensitive enough that we preemptively scale as traffic "appears" to be increasing. So we add 5 EC2 instances, not at crisis levels of traffic, but "Hmmm...I feel a tingling in my extremities". We then remove one instance at a time if traffic falls below "Not doing anything" levels. The time between scaling actions is 15 minutes. Since ASG's go to remove older instances first, we don't end up getting charged a full hour for instances that are up for less than an hour that often.

Admittedly, our web traffic is very US Business Hours centric and peaks predictably between 3 and 4 in the afternoon.

Also, we're operating more at a scale of $40,000/month for peak traffic capacity 24/7 and $25,000/month once I got autoscaling worked out. So...yeah. I guess the scale for savings matters. :-)