Would you be interested in a blog post about it? I'm a sysadmin/devops with 12 years in, and build stuff like this on a daily basis. I wouldn't mind sharing how the sausage is made.

I'm a PHP dev working with Apache on a daily basis, but besides .htaccess and some minor changes I can't really do much. I would love an article which explained in a simple way how to scale your server and really debug problems with it.

Last week, I literally spun down an EC2 instance and signed up for Digital Ocean because I couldn't figure out how the hell to make stuff work on EC2, but have lots of VPS experience. As a dev and not a sysadmin, it's much easier to go with what you know... but I want to learn.

What's the difference between EC2 and a VPS for you? Do your VPSs already have things installed or a GUI? I've used EC2 before as a single server, never scaling. The main difference was installing things that are usually pre-installed (like on Ubuntu's official desktop image). Is that it or is it more about the scaling?

And thanks for the reference to Digital Ocean. Never heard of them before. Seems great, might try using them :)

> What's the difference between EC2 and a VPS for you?

For me it that restarting an EC2 instance deletes all the local storage. I have had good success just getting a big ass VPS, and running the database locally, and pushing text backups to S3. It is trivial to manage, and in the real world, downtime is more likely to be caused by configuration wonkiness than hardware failures.

You also have to have a huge amount of traffic to overwhelm a 24core / 96GB ram server. Why not put off the managing the complexity until you really are doing 10M page views per day?

If you where having all your data deleted when you restarted your EC2 server, then something was VERY wrong. I'm not an expert, but I've used EC2 a little bit and I think I hit that exact problem.

The thing is that for whatever reason, data wasn't being written to the EBS (the virtual hard disks for use with EC2 instances) and was instead being written to the "ephemeral storage", a really big local data store that every EC2 instance has that is basically a `/tmp` directory. If the server restarts, everything in the ephemeral storage is destroyed.

EBS is a slow turd. Running a database on EBS is running a database on network attached storage. I know tons of people do this, but I have no idea why. Provisioned IOPS will work, but it is expensive.

Local storage on a $5 Digital Ocean plan will do 2000 IOPS, where as that would cost $200/month with Amazon Provisioned IOPS. I know it is not an apples to apples comparison, but it is worth thinking about. Running a database on a local SSD is a good option for many people, and it is not an option that Amazon offers.

Actually there absolutely is an instance type for this. It's the high storage io instance. Not cheap, but if you want local SSD it is definitely on the menu

I have no idea, just things don't work on EC2.

> Do your VPSs already have things installed or a GUI?

Nope. I prefer straight-up Arch linux. ssh in and go from there.

To add some concrete-ness to the mix, I was installing ejabberd. When it came time to ping the server... no response. I did the exact same steps on my Digital Ocean VPS and everything went fine. I had done whatever commands EC2 expects to open the right ports...

Ha! I knew I wasn't the only one.

I'm a large user of AWS but, in general, I never feel like I'm in a true VPS. My last experience:

Out of the sudden one of our EC2 instances could not connect to another, causing our HA solution to spun dozens of instances and eventually crash too. It was clear to me that the dest machine was behind some firewall, we went to the security group, the machine was supposed to accept any connection, from any port, any host. The instance itself had no active firewall.

Out of desperation I added the very own security group to itself. It worked for a few hours, then stopped again, I removed, it came back to work and still working (8+ months now)

This is only one of various mysterious events I've seen happening on AWS.

For EC2 security groups you have to open access to both the correct ports and protocols. Ports are a concept at layer 4 of the OSI model, while ping, or more correctly 'ICMP Echo Request', is lower down the TCP/IP stack at layer 3. So when configuring the security group, look for the option to choose Protocols, then enable ICMP :)

Thanks. This is exactly the kind of stuff that I could learn from said blog post, and why I think "EC2 == VPS" is false.

The documentation I was following made no mention of this at all.

> I have no idea, just things don't work on EC2.

A raw EC2 instance is identical to your average VPS offering. You don't have to use all the extra alphabet soup of ELB, SQS, SNS, SES, etc.

> When it came time to ping the server... no response.

What were your security group settings? ICMP (ping, traceroute, etc.) are blocked by default, you have to enable them.

> A raw EC2 instance is identical to your average VPS offering.

See, you say that, but then you later say

> What were your security group settings?

I dunno, man, I just want a server. Every other VPS provider works, and I can set up my own iptables etc.

I'm not saying EC2 is _bad_, as I'm sure things being extra mega locked down by default is good overall. But I don't think it's fair to say that EC2 is the same as a VPS.

The solution to your problem was one Googling for "AWS ping" away, and I've had more significant setup differences between two VPS providers than having to configure the AWS firewall to allow pings.

EC2 fits every definition of VPS I've ever seen.

At that point, I had already been googling for so long that giving up was the best decision. Also, when I said 'ping' I meant 'hit via a web browser' as well as ping on the command line.

Furthermore, I don't feel very comfy when doing 'sysadmin via Google,' who knows what stuff I'm screwing up?

I had never used Digital Ocean before. But getting going with them was the exact same as my previous Linode, Rackspace Cloud, prgmr.com, and every other VPS provider I have tried. I don't need to install special command line tools, or set up security groups, or generate .pems... I ask for a server, they send me an email with a password, I log in, change it, and set up my ssh prefs. Super easy, using the same stuff I use everywhere else.

The big difference between traditional VPS and IaaS services like EC2 (or Google Compute Engine, etc.) is that the latter has dynamic scaling (and a pricing model built around the assumption that you will use dynamic scaling) as a core feature; for traditional VPS-style work on an IaaS, you are likely to end up paying a premium for flexibility you aren't using (and possibly dealing with some attendant management complexity from the same source), but, other than that, IaaS should be a complete substitute for VPS.

I would absolutely love this. Can't get enough solid sysops info - as a dev I find the docs, tutorials, etc out there fairly esoteric. Anything written recently and actually finished helps a ton : )

I have quite a bit of devops experience myself, especially on Amazon Web Services and their CloudFormation service combined with Chef. I would be very interested in a blog post (or a series, go nuts!) and I'm considering doing a series of write-ups on our setup as well.

I would LOVE to see a detailed breakdown on how an experienced sysadmin would set something like this up. I've cobbled together systems before, but I've found that doing so in a robust way is difficult.

Hot dog! I am also interested. I hope your recipe is free of fillers and artificial ingredients. Looking forward to casing your blog site. Address?

Very interested!

Definitely interested. The less you assume about your readers the more helpful it will be to me.

I wouldn't mind it personally.

yes. very interested. very very interested :)

yes please.. ohh god yes.

Definitely!

YES!

Very interested as well!

That's actually, honestly, a bad way to do it. It's fine to pre-load an instance with source and application/package dependencies. At this point though, you really should be backing those up with it being deployed by Puppet/Chef.

Essentially you pre-run a puppet manifest and have it do all the first-run processing and then store THAT as the AMI/instance. This way you have an instance that only needs a few seconds to get itself ready while also integrating configuration management (which really you should be doing these days). Just some suggestions from a sysadmin.

Depends on how fast you want new instances to be online. Sometimes you want to respond to demand very fast, it can take a few minutes for an instance to register with ELB, if you have to download/build/configure packages prior you are just adding minutes on time. The faster you can spin up instances the higher you can run your servers as you need to tell amazon at what threshold should I spawn more instances

There are 3 general levels you can take with AMIs:

1) Vanilla instance AMI, have puppet/chef install everything for you, then fetch your app code and configure that

2) Use another tool to have your instance built with puppet/chef and then capture that into an AMI. Once it spins up it just needs to get your app code. Idea here is your services arent likely to need updated as fast as your app code.

3) Same as two, but when your production code is in release/maintenance mode, bake everything into an AMI. When you need to deploy new code, you need to create a new AMI, but you are creating an AMI with scripts so its no big deal :) All you have to do now is update your cloudformation

Well this is why I suggested pre-running with Puppet and 'install' your code and dependencies into the AMI before saving it. That way you get the best of both worlds.

Obviously if you have a setup with rapidly deployed code changes you'll want to have your puppet manifest grab the latest version before deploying live when spinning up a new instance.

I believe #3 is what Netflix does.

Puppet/Chef is overkill for a site running a config this simple.

I don't really agree with this. If you're defining a whole crazy master/agent relationship with Puppet then it's overkill for them, no doubt.

There's a reason many Vagrant boxes I see use Puppet to instantiate though: It's friggin simple to get started with. More advanced configurations might even end up requiring you to build your own modules and extended manifests, but I expect this configuration detailed in the article would probably amount to a couple hundred lines for the entire manifest.

I think this also underscores one of the things I like the most about Puppet (instead of Chef, which is a fine CM system as well); It's easy to get started with but powerful enough to get things done down the road. Honestly, there's never a better time to start working with Puppet than from the very beginning of a project!

Overkill for now. But it's best to learn the tools you will need while things are still simple. You know how things should get set up and can ensure that what you're telling Puppet/Chef to do achieves that.

If you wait until you've got a more complex configuration, you're fighting both the tool syntax and system setup requirements. Not to mention the clock that's ticking and telling you that you needed to be all up and running yesterday.

What do you recommend for simple configurations where you still want the repeatability/documentation/versioning tools like Chef/Puppet provide? Shell scripts?

You could build containers using Docker (http://docker.io) and Dockerfiles. Here's an example: https://github.com/steeve/docker-opencv

babushka - http://babushka.me/

scriptrock.com - FD: I'm one of the founders, we think of it as a executable documentation tool.

After building a half dozen one-off sites, each with a simple, single-server config, I found that going sitting down and learning Chef was helpful for jumpstarting the next project. All of the basic needs are the same: nginx, unicorn, postgresql, etc, and I was tired of reading through my jumble of system configuration notes. Yes, I could have baked AMIs and created new AMIs, but not everything was Amazon-based.

Hehe, yeah, that sentence doesn't sound particularly simple. :-)

Check out the code snippets I linked to from that blog post. It's pretty easy, I promise -- the tricky thing is just figuring out the various APIs.

I apparently need to get back to that book.

You know those days when you think you know a bit, and stumble upon someone who knows vastly more than you? Today was one of those days. It appears we're both Devops guys in Chicago; can I buy you a beer sometime?

If free beer is the price of entry here, I'd love to pick your brains as well, since we're about to run into this exact same problem. (I'm in Chicago as well)

Where do all you Chicago hackers hang out? I'm new to the city and I never see anything like a Chicago HN meetup posted here.

If you find out, let me know. I can tell you where the infosec / security researcher types are, but not where the startup people are.

Maybe we should just invent something new.

Sounds great. where do I sign up? :-)

Mail me? We should just arrange the next HN meetup ourselves. We can probably pick a better venue than the last one I went to here. :)

Email sent (earlier today).

I would love to come, just to listen and learn.

Hey, I'd love to join! I'll buy the beers if you two guys just teach me all the sysadmin stuff you know. :-D

I'd certainly be down. I love talking about this entire arena, specifically where it falls on its face.

Thanks for the post. Would you mind sharing how much you were paying Heroku per month (on average and peak usage)? Thanks again!

It's not simple, AMIs are a pretty bad idea and not something you should rely on due to the statefulness and vendor lock-in unless you can reproduce it in disparate deployment environments from scripts.

The AMIs can be imported and work on Eucalyptus private cloud. So there is no lock down: http://en.wikipedia.org/wiki/Eucalyptus_%28computing%29

It's actually a lot simpler than it sounds:

- Create a new EC2 instance (Literally click "Launch instance" and select Ubuntu. - Login and install any dependencies (sudo apt-get update && sudo apt-get my list of awesome packages i need installed) - Go back to the EC2 dashboard and click "Create AMI" which is just imaging the server. - I can't attest to the ease of ELB and auto-scaling rules as I haven't used them, but I would assume it's fairly straight forward - and there is a ton of resources on AWS to help you out :-)

Have fun!

I pitched a book to No Starch and a few others, and they all said there wasn't enough demand...

I know Packt was looking for a Chef author a few months ago. I declined to work with them, but I'm working on an outline for a Chef/DevOps book to self-publish.

I'm surprised I didn't know that; they've invited me to write a book on nearly every other technology I listed on my linkedin profile. :-/

That's why I don't buy books from Packt unless I can find good reviews of it written by members of the community around the software the book is about.

PostgreSQL 9.0 High Performance is a Packt book with excellent content, but it has multiple typos and grammar errors on every page. I'm sad to see such an outstanding book marred by bad editing.

I'm a friend and former colleague of the author, and having read the book from cover to cover I don't think it's true that it has lots of typos. I think you might be confusing it with another title.

If you think they're wrong, write it anyway and put it in the Kindle store.

That's the approach I'm taking :)

Leanpub.

Check out Amazon's Elastic Beanstalk - it handles the provisioning/deployment side of things - setup your environment with "eb start", update your app with just a "git aws.push".

Quite configurable via scripts too (install additional packages, etc). Under the hood it's basically the same - Elastic Load Balancer and EC2 instances.

I gave AEB a try with django about 3 months ago and let me say it was a major headache. Some times it would work other times the load balancer would fail other times it would just laugh in your face( there were many more but you get the point). I wasted weeks trying to get our app running, when it came down to only 3days before our demo run I made some bash and fabric scripts and all was running on multiple instances in about 1hr. Took a snapshot of our state and used that Ami for auto scaling. I myself won't be trying AEB again anytime soon.

I am working on a site right now http://makerops.com that will walk through via screencasts, texts, and interactive learning a lot of the issues described by the OP. How to interact, as a dev with various cloud APIs, to auto scale, manage configurations, etc.

AMI = ISO

Elastic Load Balancer = software based load balancer or traffic director

auto-scaling rules = if number of connections >50 or bandwidth > 100k or reponse time >10ms start a new instance and automatically update the load balancer so the new instance takes connections

Add in reasonable max-node count: automated processes that cost you money are dangerous if not governed. You don't want your provisioning tool spinning up endless amounts of nodes, all because some backend database has become slow (some new user pattern has emerged and is causing indexed queries to queue up), and all response times are getting trashed.