I always know what's on my servers all the time; I build immutable servers and throw away the old. And I'm not particularly knowledgeable or skilled at this; I just use a little good sense. I mean, your problems with devops may actually exist, but this is silly.
At Localytics, adding a new Play 2.x (our current standard) app into our new deployment infrastructure takes one command and about 120 seconds. The only part for which a developer needs devops input is the first production deploy, which require a signoff and some (automated, but privileged) IAM permissioning. Adding a new app on a framework we haven't incorporated yet (Rails is probably next) would take a few days, but needs only be done once for all developers at the company. And we know it works with our entire ecosystem.
You might have had bad experiences in the past, but you're extrapolating from them to the state of the world. It's not what you think.
Especially here: > Puppet? I am sorry but Puppet, chef and their ilk are horrible, who knows what state any machine is ever in at any point in time. humans will inevitably forget to ensure absent, or make an assumption about state in puppet land.
You setup the new cluster, switch the load balancer, tear down the old cluster.
I suggest you look into blog posts like this [if this is a pain point for you]: http://blog.codeship.io/2013/09/12/the-codeship-workflow-par...
If you run at near-max load:
If nothing else you should have a staging environment and that can be your second cluster too.
We are talking about Heroku here, they are more expensive than EC2 so you can get away with this on similar budgets too. ;)
I've worked for 2 previous startups doing this. US -> Australia -> UK makes a pretty good global coverage strategy. Once you can afford 3 people this can become a very effective way of ensuring 24 hour coverage.
Between all the countries in australasia and Europe 24 hour coverage is not that difficult. Not to mention rates for a lot of those countries are cheaper then FTE's in the US and if you bring them on as contractors, you still get to avoid having a business presence in those countries (for Tax and what have you).
How can you run 24/7 with 3 people ? In Australia that would mean you would have to be above the allowed 38 hour working week as it would be a 56 hour working week, overtime and possibly penalty rates would apply. Even if it is on call a lot of companies in Australia pay extra for that.
> Not to mention rates for a lot of those countries are cheaper then FTE's in the US
I do not know who you are employing in Australia but I would suggest that there is only about 5000 - 10000 USD p/a price difference for mid-level guys.
You wouldn't do 24x7 with 3 people, but you could do 24 hours x ~5.5 days with the rest on call. You could cover all Business hours within the week. Secondly you don't bring engineers on as employees (that requires all sorts of legal & Tax liability). You hire them as self employed contractors. This works out great for them (they can claim all sorts of things as tax write-offs for their "business") and you get to avoid dealing with the bureaucracy.
> do not know who you are employing in Australia but I would suggest that there is only about 5000 - 10000 USD p/a price difference for mid-level guys.
Avoid Sydney & Melbourne.
A lot of people will take the opportunity to do this kind of work for a decent wage ($65k in Brisbane, $55k in regional centres). I know people who do this work, move to Thailand and live in luxury on similar dollars. Brisbane is one of the Red Hat support centres so there are plenty of engineers moving on from lowish salaries offered.
Not to mention once US interest rates start going up the AUD is set to start dropping again.
Thanks, I hope I didn't sound too harsh as I just wanted to understand how you were doing this.
> You hire them as self employed contractors.
If you are an Australian company you may want to be careful with this. A contractor is not necessarily a contractor in Australia, there are some strange guidelines as to when a contractor may actually become an employee.
> A lot of people will take the opportunity to do this kind of work for a decent wage ($65k in Brisbane, $55k in regional centres).
I worked in Brisbane for 12 years and that sounds about right. Although the need for Linux admins seems to be increasing so there is some interesting salaries being offered.
> I know people who do this work, move to Thailand and live in luxury on similar dollars.
One can only dream, although I now live and travel around Asia .... I am just missing the $$$ :)
Well actually I am not doing it anymore. It turns out that the when the GFC hit the startup I was working for went under. So did a lot of places and I had trouble finding work. Now I have more family commitments I am a working peon for a small digital marketing company.... but there are days I wish I was back doing that kind of work again.
> If you are an Australian company you may want to be careful with this. A contractor is not necessarily a contractor in Australia, there are some strange guidelines as to when a contractor may actually become an employee.
Yeah this really only works if the contractor is working for a non-australian company. Then they are liable for their own super / tax / etc. Its a bit of a hassle, but a good tax accountant will get it sorted and reduce your liability at the same time. Contractor gets to write off their home (where they do the work), their tech, and more. Government hasn't really cracked down on it, because its an export business and your helping the economy.
>I worked in Brisbane for 12 years and that sounds about right. Although the need for Linux admins seems to be increasing so there is some interesting salaries being offered.
Unfortunately not as much as I would like... seems between a lot of "cloud first" outsourcing and the fact that linux / unix roles tend to be in HQ (i.e. in Sydney or Melbourne for aussie companies) the linux roles tend to be limited.
On the other hand PHP/Web/.Net developers with linux knowledge is in demand. (Quite strange the .Net developer with linux server experience, but just check seek.com.au and you will find them).
Brisbane is and it appears always will be a MS town. 80% of systems roles are Windows/Cisco/VMware focused. Smaller guys tend to be more Linux / Mac OS based, but then you end up with hybrid roles where your also 15 other roles.
1.) Downtime - You seemed to gloss over my primary point. The downtime today could have been completely avoided if Heroku did not opt to do a 2pm EST scheduled maintenance. This was a very large risk they took and it blew up in everyone's face.
As far as resolving downtime. I stand by my original statement that a PaaS like Heroku cannot recover as fast on average as another solution. They have too many customers that they have to get up and running.
5) Buildpacks - I've been fielding this one on Twitter all day and am considering writing another blog post. I didn't really give much weight to this one and that opens it up to attack pretty easily. Here is my point:
Buildpacks in general do not suck. Heroku's implementation of Buildpacks sucks. Customizing Heroku buildpacks is a nightmare. Maintaining said fork is nearly impossible. There is no versioning of the buildpacks. There is no way to know if/when slugs are updated and my fork is no longer useable or needs to merge in upstream changes. Getting anything into the official buildpacks is a political nightmare. Adding CLI tools not in the heavily stripped down Ubuntu AMI is a gigantic pain in the ass.
But if that is not enough for you I'll end this point with David Dollar who did a ton of work while at Heroku with its buildpacks https://twitter.com/ddollar/status/481201978391801857
re: puppet - I'm not sure why you're hammering me for Puppet as I never mentioned it. You seemed to assume that this is what we will use.
3) Pricing - I dismiss most of your argument unless we're talking about Heroku's database hosting. Their DB hosting is awesome and top-notch. Their app hosting has become a less than stellar. I don't even know what I'm paying for anymore. To me managed devops means I don't ever have to deal with pager duty or I can just hand customers over to someone after we launch their product. That's not Heroku, not by a long shot. At best we're getting security fixes and system updates applied. At worst we are paying a premium on resold EC2 instances for a deploy shell script that has been freely available for a while now.
I don't know about you but I don't really spend that much time swapping the underlying language. At least not without some heavy consideration, what we go with is usually thought out from the start of the project. To me, I'm not interested in the least in being able to swap out MRI for JRuby with a change of a line. In nearly 10 years of professional Rails development I have maybe had to do this twice. Upgrading Ruby is a different story, but that is also not something that should be done lightly.
Today's post was a culmination of years of frustration with the platform. I've had it, we're done with it. Moving on. Good riddance.
For heroku it is tougher, they are the app that runs other apps, so bugs have a much larger impact. I would hope they have post mortem's and take actions to ensure this doesn't happen again. IE why wasn't this caught in a staging environment?
I agree that heroku totally fucked up, I would feel terrible if I was the engineer that fucked this up (as they should). However I think you are throwing the baby out with the bathwater.
I have seen outages last 1 hour + at big startups in the valley, it happens. I am not saying that it is right, I am saying that it is very difficult to do right, and very few companies are able to do it. It generally takes quite a few people to do it and far too often I have seen it come at the cost of being able to move quickly. Not excusing heroku for making these mistakes, I think it blows pretty hard. I just think there is no good alternative.
I believe that you are seriously underestimating the amount of effort required to deliver 99.9% reliability. I also believe you are underestimating how much heroku does for you on a git push if you think it is a bash script that has already been written. If that were true we would all have Heroku behind the firewall.
5. Heroku's implementation of buildpacks on their stack sucks? They have the only real implemenation of buildpacks? I agree forking/maintaining the fork sucks, I think there should be an easier way to install apt-get dependencies (ENV var).
Alternatively you can use: https://github.com/ddollar/heroku-buildpack-multi which supports versioning and chaining. You just make a buildpack for your custom bits, and install it before your ruby buildpack
I agree I don't often switch between ruby interpreters, however I do frequently upgrade MRI ruby versions, and the fact that it as easy as changing one line has saved me a lot of time. IMO this is much easier than any alternative I am aware of.
I have written a few buildpacks and I have found buildpack multi to be pretty composable.
I agree Dockerfiles are a decent alternative, however the part missing is a scheduling/discovery layer. Mesos docker integration is coming along via the folks at mesosphere but you still need to wire up the bits and pieces yourself.
3. I agree it is expensive, I think it is cheaper than a full time ops person to a pretty decent scale, and I think you need a lot more than 1 ops person to get close in terms of SLA, and featureset.
Honestly I think you just traded one set of problems for another. Who is going to get paged when both the vm's hosting a service die? Who is going to get paged when you run out of disk space? Co-location of apps? Hosting multiple ruby versions on all your hosts?
I don't fully appreciate your model but IMO it sounds like you have a client expectations problem.You should set the deliverable as a deployable app with the target as heroku, if they want to pay more for a VM they can pay more but they should understand the tradeoffs. If they want their own custom maintained servers or pay for you to maintain them, but it ain't free. It will cost you time and hopefully their money.
I don't swap language often if ever, however I do like to use more than one language depending on the problem I am solving.
1) Downtime.
This is such a huge fallacy, if you think doing this yourself means no downtimes think again. If you had a top notch ops team you are still going to have outages in the middle of the business day. This team of ops guys you hire, guess what? They are not going to be deploying at 1am every day, that is not sustainable. They are going to deploy in the middle of the day, and guess what, they will make a mistake. And guess who notices outages first most times? Customers.
Heroku's uptime is typically in the 99.9%'s, I don't think you can do the same with one ops guy, maybe you will get lucky, however it is more likely that you won't.
5) Buildpacks
Build packs are an AMAZING abstraction. Just think about how you would solve the following without buildpacks:
5.1) Install libgeos on all the hosts running a specific application?
5.2) Create and scale new worker instances on demand?
5.3) How do you setup logging for all these servers?
5.4) Health checks for these services (tcp, http)
5.5) Alerting and automatic failover
* pssh? That will work fine until you spin up a second server and forget all the incatations to get your software up and running. * Puppet? I am sorry but Puppet, chef and their ilk are horrible, who knows what state any machine is ever in at any point in time. humans will inevitably forget to ensure absent, or make an assumption about state in puppet land. Nevermind the slow feedback loop in puppet land. Oh hey you want to change all the servers running X to now run Y instead, good luck with that. * Mesos? Mesos is the best open source scheduler out there, Marathon makes it easy to deploy to and there are scripts to run easily on ec2. However if you want to install native dependencies (libgeos etc) you will still need to do one of the above things. * Docker? Docker is an amazing way to package your app up and ship it to run on Mesos, however how do you package docker apps and ship them? Guess what? Buildpacks.
Now consider this on Heroku (5.1) Add the https://github.com/ddollar/heroku-buildpack-multi (5.2) Change your procfile (5.3) There are plugin's for that
Oh hey you can also write your own language, write a buildpack for it, and deploy it to production without ever having to get an ops person anywhere to setup a server for you. I am sorry but that is nothing short of amazing.
2 and 4)
I agree 100%, it sucks to feel powerless and have no insight as to when things will be resolved, however if you work in any decent size org ops things will go down, and you will also have to wait to see any resolution. I agree it feels better to see people working busily in your office because you can see the sense of urgency. I
3) Pricing:
In regards to pricing: yes Heroku is very expensive. However for small and medium teams it is much cheaper than hiring an ops person. You also get an economy of scale when you use Heroku; Think of all the things heroku takes care of for you: automatic failover on hardware/software(segfaults)/vm issues. Disks filling up on some host (it is embarrassing how many times I have seen something as simple as this cause issues). Agility; You need to switch MRI out for JRuby or rubinius, no problem, just update your ruby "version" in your Gemfile. The Heroku platform is superior in almost every way for running small to medium size applications then any in house cooked alternative.
I am very familiar with the alternatives out there, I have contributed to dokku's buildpack repo https://github.com/progrium/buildstep, and have patched docker.
There is no good behind the firewall/on my vm/hardware alternative for heroku atm. People are making progress but you won't really get anything as full featured as heroku.
Best of luck to you Brian.