"On top of that, by using such a specialized system to run your apps, you lose all the flexibility of having a full linux OS to troubleshoot and debug from. You now have to rely on them building on all the components that already exist in regular Linux world, like debuggers, tracers, sniffers, profilers, etc. You'll have to slip all that into your application deploy to troubleshoot a weird one-off bug."
CoreOS is a full fledged linux, and since applications are running in containers, there is no reason you couldn't use debugging tools on the host.
Most production services have strict controls anyway, so it isn't like it is common practice to log in to a production database server and do apt-get/yum install gdb and start banging away.
From what I understand about LXC, you have to create a chroot environment for your service to run in. This means installing applications in the chroot environment in order to use them from the application. For various types of debugging/troubleshooting, this may be necessary, as the resources of the environment and its behavior may be (read: are guaranteed to be) different from that of the host OS.
I don't know what kind of environments you work in, but "strict controls" go out the window when the production site is randomly going down and you're losing millions of dollars in revenue. When all hell breaks loose, you dig in your heels and debug the app server while it is crashing, with a developer sitting next to you and three fuming managers behind your chairs. In this scenario i'd rather have a plain old fat-ass Linux distro than a clunky "minimal" container manager.
This may happen more often than you think. There is a technique popularized by some MySQL hackers at Facebook called "poor man's query profiling" that uses gdb to (among other things) dump the the stack traces of every MySQL thread. Some awk normalizes, aggregates and sorts the traces.
I often do this when I encounter a badly flailing or completely wedged MySQL daemon. It's a good way to see inside MySQL, and if a bunch of threads are all blocked on a mutex or something it's pretty obvious.
You'd be surprised. Outside of large companies with dedicated devops, it is very common to see stuff like that happening on production machines.
But that's a use case that stuff like this is ideal for: Clone the container. Install gdb in clone, and try to reproduce the problem.
However, I have found that a lot of the benefits of virtualization are often lost because of the flexibility of having a full blown Linux OS. People stick too many services on the host and/or don't know how to properly allocate resources.
I think there is something to a simplified model where you no longer have access to any host environment and can only "fill slots". In practical terms, it makes it easier to implement features, like firmware-style upgrade and rollback of the host, for the masses.
Yes, you can do this with a full blown Linux OS on the host but it requires an operational discipline that most environments don't seem to possess.
I'm not sure I buy the whole "run multiple containers on top of an EC2 instance" thing though. I understand why they're using LXC vs traditional virtualization but it seems to me like it's a case of solving all problems by adding another layer of abstraction.
There are too many different kinds of virtualization so we'll just add another layer of virtualization.
Personally, if I get time, I'm tempted to try to PXE boot CoreOS. Even better if I can do it from Ubuntu's "MAAS", as MAAS supports IPMI for powering servers up/down, and remote control management.
MAAS or another hardware provisioning layer + CoreOS + Docker starts to become very interesting.
The simplified model you're talking about is cloud computing. It doesn't matter if your hosts are virtual or not, the point is having an abstraction layer that manages resources for you so the client application doesn't have to care.
Virt comes into play when you're tailoring your servers to your application. Example: Are you really i/o bound? A thousand 1Us with 4 disks each will deliver more iops than a couple dozen beefy VM host machines, and possibly [read: probably] cheaper than a high-performance SAN. Or, do your services just need segregated resources? LXC (or openvz) will provide that regardless of your hardware, so that may be a good fit too.
The adage 'the right tool for the job' refers not to the most expensive tool, or newest, or the most flexible, or the most anything. It's just the tool that fits right. Could you use a crescent wrench to remove lug nuts from your car? Probably. But it's not the right tool.
Eventually parts get old and break down, or a flaw is found. This part needs to either be replaced or upgraded. With slotted services (CoreOS), you can replace an individual part and only affect that part of the machine, as long as it's not integral to the entire machine.
But the machine is complex. Sometimes you have to change how fast output shaft spins, or the gearing on a transmission. Or perhaps some other part of the machine's operation has to change, and that impacts this part because they connect to each other through more gears and pulleys. (comparison: API/ABI changes, database changes, network or protocol changes)
All slotted services do is provide momentary independence. They do not reduce overall support, and they only ease maintenance for that particular service. All other complex facets of server and service maintenance remain the same. The features of CoreOS - service discovery, systemd, a minimal OS, integrated deployment, etc - can all be provided with traditional linux distributions. CoreOS doesn't do anything new or difficult.
On top of that, by using such a specialized system to run your apps, you lose all the flexibility of having a full linux OS to troubleshoot and debug from. You now have to rely on them building on all the components that already exist in regular Linux world, like debuggers, tracers, sniffers, profilers, etc. You'll have to slip all that into your application deploy to troubleshoot a weird one-off bug. And forget about ever having a service contract that requires a supported OS like RHEL or Ubuntu.
This is a product designed to make them money via service and support contracts. In that sense, they may be successful. But as a sysadmin I know there's nothing this provides that I can't get from existing open source tools. Rebuilding the internet? More like repackaging.