News for hackers. Stuff that matters.
Re:Migrating from Heroku to AWS using Docker by madeofpalk on Saturday April 04, 2015 @12:00AM (#9319847)
If you're looking at doing this, especially if you're moving from Heroku and/or using Docker, seriously consider AWS ElasticBeanstalk. http://aws.amazon.com/elasticbeanstalk/

In case you don't know, ElasticBeanstalk is the AWS Heroku but _a lot_ more flexible. It's basically a layer on top of EC2 and ELB. It takes care of creating auto-scaled EC2 instances to your specification and deploying your application to it (with rolling updates!)

Out of the box, it supports PHP, Java, Python, Ruby, Node.js, .NET, Go, and Docker (so, everything else), and it's fairly trivial to set up a heroku-like git-based deploy with something like circle ci.

Re:Migrating from Heroku to AWS using Docker by philip1209 on Saturday April 04, 2015 @12:24AM (#9319892)
It's worth noting that Elastic Beanstalk just released support for multiple docker containers per instance. This takes EBS from being a novelty to being production-grade Docker orchestration. Previously, it ran a single docker container per instance which - while good for stage/prod parity and easy rollbacks - doesn't really take advantage of the core Docker advantages over a VM. Now, you can orchestrate multiple containers per host, including nice-to-haves like zero-downtime deploys, automatic load balancing, monitoring, and log aggregation.

We're using EBS at StaffJoy and in the process of switching to multiple containers per EBS host. This means that a staging environment can have cheap redundancy (e.g. for zero-downtime deploys) by utilizing the same EC2 hosts, but also that queue workers (sporadic, processor-intensive scheduling operations) can split a processor with the more memory-intensive web app.

Getting EBS to work with the private docker registry was a bit of a pain because documentation was lacking - if you're having trouble with this, feel free to ping me. I'm considering a blog post on our deployment system, including easy rollbacks.

The problems we haven't figured out yet on EBS are streaming logs from hosts to another service (e.g. loggly/logentries/papertrail for realtime production error monitoring) and monitoring deploys (specifically to send a Slack notification that changes are fully deployed and to flush the Cloudflare cache).

(Note - here EBS = elastic beanstalk, not elastic block storage)

Edit: A final note - we ended up doing a Jenkins install instead of using CircleCI for build promotion purposes. We build a container, push to the docker registry, then deploy to a stage. We have a separate job that lets us deploy to prod. However, there's no rebuilding - it's just a button that takes a build number and tells EBS to deploy that existing build to the master cluster. For rollbacks - you just enter a prior build number and a prior build immediately starts rolling back We couldn't replicate this stage->master promotion/rollback in CircleCI.

Edit2: We chose to build the image ourself and register with the private registry instead of having AWS build it because we run our unit tests within the built container, so we can guarantee that the image that passes tests is the exact same as the one deployed to production. There was some concern that the image hash could differ - e.g. due to a pip download failure - if AWS built the image separately.

Re:Migrating from Heroku to AWS using Docker by kudu on Saturday April 04, 2015 @12:39AM (#9319928)
> (Note - here EBS = elastic beanstalk, not elastic block storage)

I, really, really don't recommend using EBS as an abbreviation for Elastic Beanstalk, since Amazon uses the actual abbreviation canonically to refer to Elastic Block Storage. At worst, just call it Beanstalk, when the AWS context is clear.

Re:Migrating from Heroku to AWS using Docker by jdub on Saturday April 04, 2015 @01:40AM (#9320038)
The AWS abbreviation for Elastic Beanstalk is EB.
Re:Migrating from Heroku to AWS using Docker by philip1209 on Saturday April 04, 2015 @12:44AM (#9319935)
We use beanstalkd queue too (SQS gets expensive at scale!), so "beanstalk" is unfortunately also equivocal in our infrastructure. I think it's just bad naming for AWS Elastic Beanstalk overall.
Re:Migrating from Heroku to AWS using Docker by coenhyde on Saturday April 04, 2015 @12:38AM (#9319925)
You should do a blog post. I had no idea Elastic Beanstalk allows multiple docker containers per instance. This changes EVERYTHING! I would love to hear how you got this all tied together.
Re:Migrating from Heroku to AWS using Docker by philip1209 on Saturday April 04, 2015 @12:41AM (#9319931)
Here's the announcement - it's a little over a week old:

https://aws.amazon.com/about-aws/whats-new/2015/03/aws-elast...

Re:Migrating from Heroku to AWS using Docker by rdoherty on Saturday April 04, 2015 @01:03AM (#9319965)
I think he meant a blog post about your entire architecture and build/deployment would be great. And I agree :)
Re:Migrating from Heroku to AWS using Docker by shitlord on Saturday April 04, 2015 @12:06AM (#9319862)
I am wondering how difficult it would be to "push to elastic beanstalk". It would be kind of cool to have some sort of git daemon running in the background, so you wouldn't necessarily need to use a third-party CI service. You could just push to a specific git remote, just like with Heroku.
Re:Migrating from Heroku to AWS using Docker by riffraff on Saturday April 04, 2015 @12:37AM (#9319919)
you don't need a third-party CI service, you can just run "eb deploy" from your git repo

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create...

Re:Migrating from Heroku to AWS using Docker by andyfang on Saturday April 04, 2015 @12:13AM (#9319868)
That's definitely something we considered doing at DoorDash. I've heard bad things about it as you want more control over deployment/package upgrades, etc. how has that played out for you?
Re:Migrating from Heroku to AWS using Docker by brucer on Saturday April 04, 2015 @12:43AM (#9319934)
I'm also curious if any large projects use it. We tried it out for a couple weeks - it worked fine but we felt overly constrained by the configuration rules and ended up rolling our own solution.
Re:Migrating from Heroku to AWS using Docker by madeofpalk on Tuesday April 07, 2015 @12:09AM (#9331961)
We use it at Mi9 to run some of Australia's highest trafficked websites, like ninemsn.com.au (and family of new sites) and jumpin.com.au. Also Stan.com.au (Australian Netflix competitor) runs their micro service architecture on Elasticbeanstalk