News for hackers. Stuff that matters.
Re:Containers, Not Virtual Machines, Are the Future Cloud by lsc on Tuesday June 18, 2013 @11:01AM (#5899592)
Having sold both containers and VMs to untrusted users? I disagree. Now, my container experience is old, and containerization technology is improving all the time, so over time, what I say here becomes less true, but that's the thing; I'm willing to pay a fairly high cost in (cheap) ram for relatively small gains in compartmentalization. (which is to say, I'm willing to spend a fair bit of money to make my pager go off less.)

See, the expensive part? it's not the ram, it's sysadmin time. That's why VPSs are killing shared hosting; it's actually easier to manage a bunch of VPSs than a bunch of people with shell accounts on the same box. Same goes for all the containerization technologies I've tried; it's easier to just throw xen/kvm on there and pay for the extra ram than to manage one user stepping on resources required by another user.

For me? that was the beautiful part of switching from jails to Xen; suddenly, I could just ignore customers who tried to burn all the cpu and ram they could. Now, if I can figure out how to do the same with disk? I'll be a very happy man.

That, at least, is where I'm putting my money; in more ram.

Re:Containers, Not Virtual Machines, Are the Future Cloud by ChuckMcM on Tuesday June 18, 2013 @03:29PM (#5901431)
So one of the projects I got to particpate in at Google before I left were efforts at containerizing disk I/O such that each container could get a strong guarantee of x% of the disks available bandwidth. That resulted in a number of really useful changes to the disk scheduler code. I expect that the code has been contributed back by this time but I've not been following it closely.

Like the author I think containers are a much better answer than VMs. That is particularly true in a "web 2.0" world. I wrote up a bit about that on http://ops.blekko.com. In an architecture where the "app" is effectively a schema across a bunch of platform services you can achieve much higher machine utilization with containers.

At some point I hope the team at Google writes up their results for publication.

Re:Containers, Not Virtual Machines, Are the Future Cloud by davidstrauss on Tuesday June 18, 2013 @01:36PM (#5900723)
> Now, if I can figure out how to do the same with disk?

Hi, I'm the author of the article.

You should check out cgroups. You can isolate disk access between services and containers using block I/O shares with cgroups.

The way shares work in cgroups is that there's unlimited access until the resource comes under contention. Then, the kernel schedules activity based on which cgroups have shares remaining. This allows scheduling to be "fair" in the sense that no cgroup always wins; access just becomes proportional.

We use block I/O shares (and other cgroup methods) for isolating containerized MySQL instances on Pantheon, the platform I work on as CTO.

Re:Containers, Not Virtual Machines, Are the Future Cloud by boryas on Tuesday June 18, 2013 @07:13PM (#5902647)
Doesn't the blkio cgroup not know anything about buffered writes? Or do the shares work differently and that limitation is only for accounting?
Re:Containers, Not Virtual Machines, Are the Future Cloud by davidstrauss on Tuesday June 18, 2013 @07:35PM (#5902739)
Yes: "All the buffered writes are still system wide and not per group. Hence we will not see service differentiation between buffered writes between groups." [1]

[1] https://www.kernel.org/doc/Documentation/cgroups/blkio-contr...

Re:Containers, Not Virtual Machines, Are the Future Cloud by lsc on Wednesday June 19, 2013 @12:24AM (#5903739)
huh. interesting. this was the primary reason jails sucked so hard for me. The heavy users would push the light users pagecache out of ram.

However, using xen or kvm solves that problem, by giving each guest their own ram that nobody else can fuck with. I wonder if then using cgroups for the remaining writes would work well? seems like the 'shares' approach might work better than the 'priority' approach that ionice uses.

Re:Containers, Not Virtual Machines, Are the Future Cloud by davidstrauss on Wednesday June 19, 2013 @04:21AM (#5904361)
> However, using xen or kvm solves that problem, by giving each guest their own ram that nobody else can fuck with.

It gets hard to say whether a shared page cache is a good thing or not, even though it may be unfair. I say this because I/O bandwidth getting exhausted is a huge issue.

For example, using cgroups to limit memory allocations for groups or processes seemed like a great way to fairly distribute memory. But, doing so forced such cgroups into swapping when they tried to exceed their limits, even when there was available memory on the host system. The swapping was so bad in terms of saturating disk I/O that we had two choices to maintain quality of service: (1) set the hard limit and OOM kill (or equivalent) within the cgroup when it gets exceeded or (2) not treat it as a hard limit and monitor usage separately. We chose the latter.

So, I honestly wonder, is it better to enforce separate page caches in the cause of fairness, even if it results in less efficient disk I/O? Or is it better to have a unified page cache and dedicate system-wide resources to increasing effective disk I/O bandwidth? (Do we focus on slicing the pie more fairly or increasing the size the pie while cutting sloppily?)

Re:Containers, Not Virtual Machines, Are the Future Cloud by lsc on Wednesday June 19, 2013 @09:06AM (#5905264)
>Do we focus on slicing the pie more fairly or increasing the size the pie while cutting sloppily?

In my case, (well, really in the flat-rate multi-tenant case in general) fairness is what I care about, more than overall efficiency. From a business perspective, it's okay to not be all things to all people, if you are a cheap-ish flat-rate service, it's okay if your heavy users don't get as much performance as they'd like, as long as the reasonable usage customers still feel like they are getting what they are paying for.

My core customer, the hobbyist who wants something like a shell for IRC idling, personal mail server, DNS for personal domains and a place to experiment or run a development project... generally buys enough ram to cache all the disk they normally access. I mean, 512MiB goes a long way for your average unix sysadmin. So even if the disk is getting thrashed by heavy users, light users? once their pagecache is warm, they have a fairly responsive system for everyday sorts of things. Under Xen or something else that doesn't share pagecache? once the pagecache is warm, it stays warm. They can login a week later (assuming they aren't running a bunch of background stuff that's reading/writing disk and churning pagecache) and their /etc/shadow is still in pagecache, and the login is pretty fast.

Back when I was using FreeBSD jails? if the user hadn't logged in for a few hours, the /etc/shadow in their jail had been flushed from cache, and they had to read it from disk again. They were getting terrible service.

From a economic standpoint? in general, if you have a flat-rate service, your light users are where the money is. They pay just as much as your heavy users (as it is flat rate) and they use fewer resources. And really, I think it's fair that if you are a light user and you are paying just as much as the heavy user, you should promptly get your resources on the occasions that you do need them. Heavy users on flat-rate services are going to have less than perfect experiences, and this is fair too, I think, so long as expectations of service level were set ahead of time. Heavy users use flat-rate services because it's cheaper than pay per use services, usually... but the downside of that is that they then only get as much resources as they can get without impacting the light users.

Now, in a pay per use model? the opposite is true. In a pay per use model (like, say, google app engine) the heavy users are the real customers. The light users are sales prospects. So yeah; in that case, you focus on the heavy users (and charge accordingly.)

Personally, I believe this is why shared hosting is seen as so far inferior to platform as a service; shared hosting is usually billed flat-rate, so the service provider really only has incentive to look after the light users; they are better off if the heavy users go elsewhere. (and really, for five bucks a month, what do you want?) - whereas platform as a service usually means you get billed per use; in that case, it makes a lot of sense for the provider to focus almost all it's energy on making things better for the heavy users. Of course, they also charge those heavy users accordingly.

Re:Containers, Not Virtual Machines, Are the Future Cloud by csense on Tuesday June 18, 2013 @05:52PM (#5902303)
Parent's business provides virtual servers to the general public (disclosure: I'm a customer). With Xen, guests can run any distro kernel, self-compiled kernel with custom patches, or even BSD -- any guest kernel with Xen patches should work. The guests can update their kernel on their own, and the only the guest that's updating their kernel needs a reboot -- the host, and guests which don't change kernels except during major upgrades, can have months or years of uptime.

So if you have a business where some of your customers want custom, bleeding-edge kernels, and others mainly value stable systems with high uptimes, and everyone wants to run a different distribution's kernel, then containers aren't for you.

I guess I'm trying to say that VPS will always have a niche in the market. The parent lives in that niche. Containers offer fundamental resource-efficiency advantages and may capture large markets from Xen and Xen-like technologies over the next few years, but I disagree with the article's implication that Xen will eventually become extinct. As the parent notes, RAM is cheap, and the difference in costs between container and Xen will give the former an advantage, but never be big enough to drive the latter completely out of business, since the difference is small (on an absolute scale) and Xen can do a few things that containers can't.

Re:Containers, Not Virtual Machines, Are the Future Cloud by porker on Tuesday June 18, 2013 @01:31PM (#5900694)
> See, the expensive part? it's not the ram, it's sysadmin time.

And that's why I like containers, because the sysadmin time of updating and patching all VMs I own (and manage for friends/clients) is huge.

Yes, I should probably use chef/puppet. The potential incompatibility issues with automated VM upgrades are probably the same as upgrading the container's OS. But mentally it feels more risky.

Re:Containers, Not Virtual Machines, Are the Future Cloud by dspillett on Tuesday June 18, 2013 @01:49PM (#5900812)
> because the sysadmin time of updating and patching all VMs I own

There is a key difference between your circumstance the person you are replying to: you are managing a farm of your own VMs so that admin time is your's to find. If the VMs are sold to thers so you are just hosting them, then the VM updates are someone else's problem and the admin time that becomes your problem is dealing with people whose VMs that are (accidentally usually, but not always) abusing shared resources. This is something you don't have as much hassle with when all the VMs are your's: you have presumably planned the likely resource needs carefuly and if something out of the ordinary happens you won't answer back to yourself like someone who has bought a cheap VM would answer back to you when you tell them they seem to be doing something wrong.

Selling containers/VMs to others is a very different kettle of fish to running you own private farm.

Re:Containers, Not Virtual Machines, Are the Future Cloud by davidgerard on Tuesday June 18, 2013 @02:08PM (#5900943)
If you buy a VM on Amazon, you get to do your apt-get updates yourself. And yes, managing it is a PITA.
Re:Containers, Not Virtual Machines, Are the Future Cloud by susi22 on Tuesday June 18, 2013 @02:53PM (#5901216)
Shouldn't be a PITA. Look into Vagrant, ansible, salt stack, chef, puppet etc.
Re:Containers, Not Virtual Machines, Are the Future Cloud by bmullan on Wednesday June 19, 2013 @08:57AM (#5905212)
Fabric also works great.

If you ice ubuntu... JuJu is terrific for either VM or container use. I have used it to deploy all of open stack services into a single "super size" AWS EC2 VM where the open stack was configured to use LXC as its virtualization. Worked fine for what I wanted. To back up or clone that open stack cloud only required me to snapshot that single EC2 instance.

Re:Containers, Not Virtual Machines, Are the Future Cloud by porker on Tuesday June 18, 2013 @04:31PM (#5901869)
Does vagrant apply in this situation? I thought it was only for dev machines
Re:Containers, Not Virtual Machines, Are the Future Cloud by wahnfrieden on Tuesday June 18, 2013 @11:04PM (#5903521)
Flying_Dwarf: Your account is hellbanned (seems to be for talking about hipsters in SF), just FYI.

How does salt compare with puppet?

Re:Containers, Not Virtual Machines, Are the Future Cloud by hoov on Wednesday June 19, 2013 @01:21AM (#5903852)
Roll your servers more often. If you have autoscaling set up, you should be good to go. Yes, this is harder for infrastructure servers, but you should automate the creation of those as well.
Re:Containers, Not Virtual Machines, Are the Future Cloud by Theory5 on Wednesday June 19, 2013 @11:30AM (#5906086)
Not really. I don't do any customer-oriented things on EC2 so my experience may differ, but to manage my VM's I just setup a bunch of cronjobs to update everything, reboot every so often and start my programs and services after each reboot.
Re:Containers, Not Virtual Machines, Are the Future Cloud by gwgarry on Tuesday June 18, 2013 @08:50PM (#5903018)
apt-get update; apt-get upgrade after you try that on a clone machine. it's not a PITA unless you're super lazy which apparently many people are.
Re:Containers, Not Virtual Machines, Are the Future Cloud by fragmede on Tuesday June 18, 2013 @08:59PM (#5903049)
Easy for one machine. Now do it for 5, or 10, or 100.

Remember that those machines all have slightly different software on them, so it's not simple to automate. Unless you're manage systems with something (eg puppet) it's not just a PITA or a question of laziness - it's a real task that takes up hours, not a quick 15-minute job you run over a coffee break, and that's assuming no breakage.

Re:Containers, Not Virtual Machines, Are the Future Cloud by porker on Thursday June 20, 2013 @03:40AM (#5910745)
That's fine so long as you have a full test suite to exercise the app - catch small mail server configuration changes, apparmor and MySQL (we got bitten by that when upgrading to MySQL 5.5 from 5.1) et al - it's not a matter of saying "hey this looks like it's running" but verifying there's no regressions either.

Mind you containers don't help with this either...

Re:Containers, Not Virtual Machines, Are the Future Cloud by bmullan on Wednesday June 19, 2013 @08:47AM (#5905162)
And at least with ubuntu you can pass user data to a new container just like on AWS EC2 to initialize a new container with specific apt-get installs etc
Re:Containers, Not Virtual Machines, Are the Future Cloud by jacques_chester on Tuesday June 18, 2013 @08:56PM (#5903035)
> That's why VPSs are killing shared hosting; it's actually easier to manage a bunch of VPSs than a bunch of people with shell accounts on the same box

I came to the same conclusion nearly 5 years ago:

http://clubtroppo.com.au/2008/07/10/shared-hosting-is-doomed...

I overreached though. The thing that keeps shared hosting alive is the gigantic momentum of PHP apps like Wordpress, various bulletin boards etc.

Re:Containers, Not Virtual Machines, Are the Future Cloud by lsc on Wednesday June 19, 2013 @12:15AM (#5903709)
Yeah. I saw that at the time, I think.

>I overreached though. The thing that keeps shared hosting alive is the gigantic momentum of PHP apps like Wordpress, various bulletin boards etc.

I think the thing that keeps shared hosting alive is that most people don't want to be SysAdmins. VPSs don't remove the need for a sysadmin's services, they just push it down to the customer.

From that perspective, shared php hosting is not only more expensive to do well (vs. selling VPSs) it provides more value to the customer (when done well) - unfortunately, market expectations right now make people expect to pay more for a VPS; the net effect being that most shared hosting services have underfunded admin departments and are thus, kinda crap.

I mean, even as someone who sells VPSs? I want the shared hosting market to be stronger. Selling a VPS to someone who is not able or willing to sysadmin a linux box is not fun or profitable. It's a bad day for the customer, and it's a bad day for me. And it's reasonable to want to not sysadmin your own server; it's a lot of work. We need some place for those people to go.

Re:Containers, Not Virtual Machines, Are the Future Cloud by andybak on Wednesday June 19, 2013 @08:50AM (#5905179)
My thoughts when reading the original piece were related to how I feel about Webfaction (shared hosting with cgroups).

The reason I have stuck with them and not shifted to a VPS is because I would need a sysadmin - and quite possibly an on-call 24/7 sysadmin at that.

Although Webfaction don't promise to fix problems at the app level - they do help and in many cases have got me out of a pickle that I would have needed a sysadmin to fix if I was on a VPS.

Re:Containers, Not Virtual Machines, Are the Future Cloud by andybak on Wednesday June 19, 2013 @08:52AM (#5905183)
EDIT - as a balance to the above statement - even with cgroups the compartmentalisation is weaker than a CPS and I do have the occasional 'bad neighbour' problem.
Re:Containers, Not Virtual Machines, Are the Future Cloud by jacques_chester on Wednesday June 19, 2013 @01:41AM (#5903886)
That's why at the time I pointed out that a new breed of apps, written to the VPS as a target, could take the administrative burden off both host and user and move it upstream to the app.
Re:Containers, Not Virtual Machines, Are the Future Cloud by lsc on Wednesday June 19, 2013 @09:34AM (#5905420)
Maybe. We have all sorts of tools that are reducing the administrative overhead of running a linux box, but they are far from eliminating that burden.

Further, most of the tools I've seen? webmin and the like? The thing is, it works great... Usually, for a while. Then sometimes it breaks, and it's a huge emergency, as the customer had it working just fine for a while; the customer was tricked into thinking they knew what they were doing, and now they've got an emergency on their hands, and they don't even have the training that installing the apps by hand forces on you.

So yeah; the thing I see displacing shared hosting? not VPSs, but another kind of fully managed service. See, as you point out, the difficulty of managing a shared hosting host goes up with the number of customers on that host in a kindof exponential way. What if you made something like shared hosting (where the user doesn't have root access, and the host is managed by full-time sysadmins) but ran one user per host, and run each host in a VM or container or something that is managed in a largely automated way?

Spend the ram/cpu overhead to better compartmentalize users and take the exponential increase in difficulty out of adding more customers to a particular bit of hardware? It's something I've thought about for a while. Problem is, the temptation to give users root on their VM would be very strong, and once that happens? well, managing thousands of hosts isn't that big of a deal... managing thousands of sysadmins? yeah. Not a job I want. Either I manage the host or you manage the host. Sharing root is a difficult, high-touch thing that requires a lot of communication.

Edit: I believe that many of the platform as a service providers operate like this (e.g. your stuff runs in one or several VMs that are not shared with other users, but that you don't have root on.)

Considering how cheap platform as a service is for the light users, I wonder how that model hasn't destroyed the flat-rate 'shared hosting' model? Perhaps one of the platform as a service providers needs to offer a flat monthly plan (perhaps that has the option of degrading /or/ interrupting service when usage exceeds the agreed-upon threshold, sort of in emulation of the "unlimited" providers.)

Re:Containers, Not Virtual Machines, Are the Future Cloud by jacques_chester on Wednesday June 19, 2013 @08:55PM (#5909522)
My model isn't "install some click-and-point admin" tools. As you point out, it's been done.

It's that if you have an app that targets bare virtual metal, then you can push a lot of admin problems upstream to the app developers.

In particular: updating. If a single app controls the whole VM, you can drastically improve the likelihood that a single update mechanism will work properly. You can even automate it.

Similarly: configuring components. Right now the lowest common denominator software environment that is targeted is shared hosting. In shared hosting I cannot adjust the database, the web server and so on. So no account is made of how to make self-adjustability a part of the architecture.

And so on.

My point is that we should stop thinking about apps that we install on linux that are on VMs vs apps that are installed on linux in shared hosting environments. It's that the app should be conceptually be all that runs on the VM. That nobody installs "the stack", they install "the app".

I think it would still be a distinct niche apart from managed systems.

Edit: a re-reading of your comment suggests that, I think, we are groping towards a common concept, but from slightly different directions. If you're interested, I'll email a project proposal I wrote in 2010 about this very topic. See my profile.

Re:Containers, Not Virtual Machines, Are the Future Cloud by lsc on Thursday June 20, 2013 @12:46PM (#5913093)
>In particular: updating. If a single app controls the whole VM, you can drastically improve the likelihood that a single update mechanism will work properly. You can even automate it.

This is interesting. I mean, if you can get it to work. I'm bringing up the point and click admin tools, though, because that's the best comparison I can come up with that actually exists that I've seen.

The thing of it is, I think that most things do eventually break. (I mean, like the kids in this thread talking about how easy an 'apt-get update; apt-get upgrade' is - it works great most of the time, then it breaks something horribly.)

There are things you can do to reduce the chance of that breakage, but eh, at some point you have to login and fix the goddamn thing.

Not that this breaks your model; I mean, if you are distributing, say, a blogging application; my dad is paying you $5/month and giving you login credentials on a standard VM of some type, he installs using the kernel/script you gave him, and the system uses your repos to upgrade nightly. If something does break horribly, you can login and fix it.

I still think that model is easier if you include the VM in that $5/month, and you run that VM on something you control.

Re:Containers, Not Virtual Machines, Are the Future Cloud by jacques_chester on Thursday June 20, 2013 @08:25PM (#5915766)
That said, there are millions of copies of unpatched systems running on VPSes across the internet. I think that the lesser evil is automation. Possibly the upstream can borrow from continuous delivery and only "announce" the upgrade progressively, testing as it goes.

I'd also be interested in the upgrade system itself. Systems which have atomic snapshotting would be more un-borkable, for example.

Re:Containers, Not Virtual Machines, Are the Future Cloud by jauer on Tuesday June 18, 2013 @12:30PM (#5900270)
SmartOS has disk IO throttling for KVM guests.
Re:Containers, Not Virtual Machines, Are the Future Cloud by lsc on Tuesday June 18, 2013 @12:57PM (#5900461)
and you can throttle with ionice on xen running a Linux dom0[1]. It works pretty great if you have some user who is going overboard and you want to make it suck for just them and suck less for everyone else.

It doesn't work like the CPU scheduler; which is to say, I can't set ionice on all the guests and expect good performance on the light-use guests while a heavy-use guest is thrashing the disk. (I can expect the good performance for the light-use guests when the resource in contention is CPU time.) It's possible that either ionice isn't a good enough tool, or that I don't know enough about it... but I think it's far more likely that the variable nature of disk throughput makes balancing requests harder... (e.g. how many IOPS a second I get varies a lot from second to second, based on where the sectors I want to operate on are in relation to the read/write head. How much disk bandwidth I get varies even more.)

I am pretty curious about how much this situation will change with SSD storage.

[1]http://wiki.prgmr.com/mediawiki/index.php/Chapter_7:_Hosting...

Re:Containers, Not Virtual Machines, Are the Future Cloud by nitrogen on Wednesday June 19, 2013 @12:49AM (#5903790)
Maybe with spinning disks, instead of/in addition to allocating IO operations or bandwidth, you could allocate seek distance.
Re:Containers, Not Virtual Machines, Are the Future Cloud by lsc on Wednesday June 19, 2013 @09:15AM (#5905319)
My understanding is that the cyl/head/track info the operating system gets is... pretty far abstracted from reality by the disk firmware, making that difficult to do in any but a very rough way. By 'very rough' I mean that in the normal case, what the OS sees as sequential sectors are sequential on disk, modulo the bad block remapping... but, from what I understand, going further, e.g. Is sector 5557 closer to 6669 or 3331? is all but impossible to answer from the OS. I mean, most of the time we assume that 6669 is closer as the difference between 6669 and 5557 is smaller than the difference between 3331 and 5557, but because the slow part of the disk is usually the rotation and not the head seek, this is not always or even usually the case.
Re:Containers, Not Virtual Machines, Are the Future Cloud by davidstrauss on Wednesday June 19, 2013 @04:25AM (#5904376)
Especially once you start using hardware RAID controllers, the physical overhead of running I/O commands is pretty abstracted away from the kernel.