Like the author I think containers are a much better answer than VMs. That is particularly true in a "web 2.0" world. I wrote up a bit about that on http://ops.blekko.com. In an architecture where the "app" is effectively a schema across a bunch of platform services you can achieve much higher machine utilization with containers.
At some point I hope the team at Google writes up their results for publication.
Hi, I'm the author of the article.
You should check out cgroups. You can isolate disk access between services and containers using block I/O shares with cgroups.
The way shares work in cgroups is that there's unlimited access until the resource comes under contention. Then, the kernel schedules activity based on which cgroups have shares remaining. This allows scheduling to be "fair" in the sense that no cgroup always wins; access just becomes proportional.
We use block I/O shares (and other cgroup methods) for isolating containerized MySQL instances on Pantheon, the platform I work on as CTO.
[1] https://www.kernel.org/doc/Documentation/cgroups/blkio-contr...
However, using xen or kvm solves that problem, by giving each guest their own ram that nobody else can fuck with. I wonder if then using cgroups for the remaining writes would work well? seems like the 'shares' approach might work better than the 'priority' approach that ionice uses.
It gets hard to say whether a shared page cache is a good thing or not, even though it may be unfair. I say this because I/O bandwidth getting exhausted is a huge issue.
For example, using cgroups to limit memory allocations for groups or processes seemed like a great way to fairly distribute memory. But, doing so forced such cgroups into swapping when they tried to exceed their limits, even when there was available memory on the host system. The swapping was so bad in terms of saturating disk I/O that we had two choices to maintain quality of service: (1) set the hard limit and OOM kill (or equivalent) within the cgroup when it gets exceeded or (2) not treat it as a hard limit and monitor usage separately. We chose the latter.
So, I honestly wonder, is it better to enforce separate page caches in the cause of fairness, even if it results in less efficient disk I/O? Or is it better to have a unified page cache and dedicate system-wide resources to increasing effective disk I/O bandwidth? (Do we focus on slicing the pie more fairly or increasing the size the pie while cutting sloppily?)
In my case, (well, really in the flat-rate multi-tenant case in general) fairness is what I care about, more than overall efficiency. From a business perspective, it's okay to not be all things to all people, if you are a cheap-ish flat-rate service, it's okay if your heavy users don't get as much performance as they'd like, as long as the reasonable usage customers still feel like they are getting what they are paying for.
My core customer, the hobbyist who wants something like a shell for IRC idling, personal mail server, DNS for personal domains and a place to experiment or run a development project... generally buys enough ram to cache all the disk they normally access. I mean, 512MiB goes a long way for your average unix sysadmin. So even if the disk is getting thrashed by heavy users, light users? once their pagecache is warm, they have a fairly responsive system for everyday sorts of things. Under Xen or something else that doesn't share pagecache? once the pagecache is warm, it stays warm. They can login a week later (assuming they aren't running a bunch of background stuff that's reading/writing disk and churning pagecache) and their /etc/shadow is still in pagecache, and the login is pretty fast.
Back when I was using FreeBSD jails? if the user hadn't logged in for a few hours, the /etc/shadow in their jail had been flushed from cache, and they had to read it from disk again. They were getting terrible service.
From a economic standpoint? in general, if you have a flat-rate service, your light users are where the money is. They pay just as much as your heavy users (as it is flat rate) and they use fewer resources. And really, I think it's fair that if you are a light user and you are paying just as much as the heavy user, you should promptly get your resources on the occasions that you do need them. Heavy users on flat-rate services are going to have less than perfect experiences, and this is fair too, I think, so long as expectations of service level were set ahead of time. Heavy users use flat-rate services because it's cheaper than pay per use services, usually... but the downside of that is that they then only get as much resources as they can get without impacting the light users.
Now, in a pay per use model? the opposite is true. In a pay per use model (like, say, google app engine) the heavy users are the real customers. The light users are sales prospects. So yeah; in that case, you focus on the heavy users (and charge accordingly.)
Personally, I believe this is why shared hosting is seen as so far inferior to platform as a service; shared hosting is usually billed flat-rate, so the service provider really only has incentive to look after the light users; they are better off if the heavy users go elsewhere. (and really, for five bucks a month, what do you want?) - whereas platform as a service usually means you get billed per use; in that case, it makes a lot of sense for the provider to focus almost all it's energy on making things better for the heavy users. Of course, they also charge those heavy users accordingly.
So if you have a business where some of your customers want custom, bleeding-edge kernels, and others mainly value stable systems with high uptimes, and everyone wants to run a different distribution's kernel, then containers aren't for you.
I guess I'm trying to say that VPS will always have a niche in the market. The parent lives in that niche. Containers offer fundamental resource-efficiency advantages and may capture large markets from Xen and Xen-like technologies over the next few years, but I disagree with the article's implication that Xen will eventually become extinct. As the parent notes, RAM is cheap, and the difference in costs between container and Xen will give the former an advantage, but never be big enough to drive the latter completely out of business, since the difference is small (on an absolute scale) and Xen can do a few things that containers can't.
And that's why I like containers, because the sysadmin time of updating and patching all VMs I own (and manage for friends/clients) is huge.
Yes, I should probably use chef/puppet. The potential incompatibility issues with automated VM upgrades are probably the same as upgrading the container's OS. But mentally it feels more risky.
There is a key difference between your circumstance the person you are replying to: you are managing a farm of your own VMs so that admin time is your's to find. If the VMs are sold to thers so you are just hosting them, then the VM updates are someone else's problem and the admin time that becomes your problem is dealing with people whose VMs that are (accidentally usually, but not always) abusing shared resources. This is something you don't have as much hassle with when all the VMs are your's: you have presumably planned the likely resource needs carefuly and if something out of the ordinary happens you won't answer back to yourself like someone who has bought a cheap VM would answer back to you when you tell them they seem to be doing something wrong.
Selling containers/VMs to others is a very different kettle of fish to running you own private farm.
If you ice ubuntu... JuJu is terrific for either VM or container use. I have used it to deploy all of open stack services into a single "super size" AWS EC2 VM where the open stack was configured to use LXC as its virtualization. Worked fine for what I wanted. To back up or clone that open stack cloud only required me to snapshot that single EC2 instance.
How does salt compare with puppet?
Remember that those machines all have slightly different software on them, so it's not simple to automate. Unless you're manage systems with something (eg puppet) it's not just a PITA or a question of laziness - it's a real task that takes up hours, not a quick 15-minute job you run over a coffee break, and that's assuming no breakage.
Mind you containers don't help with this either...
I came to the same conclusion nearly 5 years ago:
http://clubtroppo.com.au/2008/07/10/shared-hosting-is-doomed...
I overreached though. The thing that keeps shared hosting alive is the gigantic momentum of PHP apps like Wordpress, various bulletin boards etc.
>I overreached though. The thing that keeps shared hosting alive is the gigantic momentum of PHP apps like Wordpress, various bulletin boards etc.
I think the thing that keeps shared hosting alive is that most people don't want to be SysAdmins. VPSs don't remove the need for a sysadmin's services, they just push it down to the customer.
From that perspective, shared php hosting is not only more expensive to do well (vs. selling VPSs) it provides more value to the customer (when done well) - unfortunately, market expectations right now make people expect to pay more for a VPS; the net effect being that most shared hosting services have underfunded admin departments and are thus, kinda crap.
I mean, even as someone who sells VPSs? I want the shared hosting market to be stronger. Selling a VPS to someone who is not able or willing to sysadmin a linux box is not fun or profitable. It's a bad day for the customer, and it's a bad day for me. And it's reasonable to want to not sysadmin your own server; it's a lot of work. We need some place for those people to go.
The reason I have stuck with them and not shifted to a VPS is because I would need a sysadmin - and quite possibly an on-call 24/7 sysadmin at that.
Although Webfaction don't promise to fix problems at the app level - they do help and in many cases have got me out of a pickle that I would have needed a sysadmin to fix if I was on a VPS.
Further, most of the tools I've seen? webmin and the like? The thing is, it works great... Usually, for a while. Then sometimes it breaks, and it's a huge emergency, as the customer had it working just fine for a while; the customer was tricked into thinking they knew what they were doing, and now they've got an emergency on their hands, and they don't even have the training that installing the apps by hand forces on you.
So yeah; the thing I see displacing shared hosting? not VPSs, but another kind of fully managed service. See, as you point out, the difficulty of managing a shared hosting host goes up with the number of customers on that host in a kindof exponential way. What if you made something like shared hosting (where the user doesn't have root access, and the host is managed by full-time sysadmins) but ran one user per host, and run each host in a VM or container or something that is managed in a largely automated way?
Spend the ram/cpu overhead to better compartmentalize users and take the exponential increase in difficulty out of adding more customers to a particular bit of hardware? It's something I've thought about for a while. Problem is, the temptation to give users root on their VM would be very strong, and once that happens? well, managing thousands of hosts isn't that big of a deal... managing thousands of sysadmins? yeah. Not a job I want. Either I manage the host or you manage the host. Sharing root is a difficult, high-touch thing that requires a lot of communication.
Edit: I believe that many of the platform as a service providers operate like this (e.g. your stuff runs in one or several VMs that are not shared with other users, but that you don't have root on.)
Considering how cheap platform as a service is for the light users, I wonder how that model hasn't destroyed the flat-rate 'shared hosting' model? Perhaps one of the platform as a service providers needs to offer a flat monthly plan (perhaps that has the option of degrading /or/ interrupting service when usage exceeds the agreed-upon threshold, sort of in emulation of the "unlimited" providers.)
It's that if you have an app that targets bare virtual metal, then you can push a lot of admin problems upstream to the app developers.
In particular: updating. If a single app controls the whole VM, you can drastically improve the likelihood that a single update mechanism will work properly. You can even automate it.
Similarly: configuring components. Right now the lowest common denominator software environment that is targeted is shared hosting. In shared hosting I cannot adjust the database, the web server and so on. So no account is made of how to make self-adjustability a part of the architecture.
And so on.
My point is that we should stop thinking about apps that we install on linux that are on VMs vs apps that are installed on linux in shared hosting environments. It's that the app should be conceptually be all that runs on the VM. That nobody installs "the stack", they install "the app".
I think it would still be a distinct niche apart from managed systems.
Edit: a re-reading of your comment suggests that, I think, we are groping towards a common concept, but from slightly different directions. If you're interested, I'll email a project proposal I wrote in 2010 about this very topic. See my profile.
This is interesting. I mean, if you can get it to work. I'm bringing up the point and click admin tools, though, because that's the best comparison I can come up with that actually exists that I've seen.
The thing of it is, I think that most things do eventually break. (I mean, like the kids in this thread talking about how easy an 'apt-get update; apt-get upgrade' is - it works great most of the time, then it breaks something horribly.)
There are things you can do to reduce the chance of that breakage, but eh, at some point you have to login and fix the goddamn thing.
Not that this breaks your model; I mean, if you are distributing, say, a blogging application; my dad is paying you $5/month and giving you login credentials on a standard VM of some type, he installs using the kernel/script you gave him, and the system uses your repos to upgrade nightly. If something does break horribly, you can login and fix it.
I still think that model is easier if you include the VM in that $5/month, and you run that VM on something you control.
I'd also be interested in the upgrade system itself. Systems which have atomic snapshotting would be more un-borkable, for example.
It doesn't work like the CPU scheduler; which is to say, I can't set ionice on all the guests and expect good performance on the light-use guests while a heavy-use guest is thrashing the disk. (I can expect the good performance for the light-use guests when the resource in contention is CPU time.) It's possible that either ionice isn't a good enough tool, or that I don't know enough about it... but I think it's far more likely that the variable nature of disk throughput makes balancing requests harder... (e.g. how many IOPS a second I get varies a lot from second to second, based on where the sectors I want to operate on are in relation to the read/write head. How much disk bandwidth I get varies even more.)
I am pretty curious about how much this situation will change with SSD storage.
[1]http://wiki.prgmr.com/mediawiki/index.php/Chapter_7:_Hosting...
See, the expensive part? it's not the ram, it's sysadmin time. That's why VPSs are killing shared hosting; it's actually easier to manage a bunch of VPSs than a bunch of people with shell accounts on the same box. Same goes for all the containerization technologies I've tried; it's easier to just throw xen/kvm on there and pay for the extra ram than to manage one user stepping on resources required by another user.
For me? that was the beautiful part of switching from jails to Xen; suddenly, I could just ignore customers who tried to burn all the cpu and ram they could. Now, if I can figure out how to do the same with disk? I'll be a very happy man.
That, at least, is where I'm putting my money; in more ram.