Re:An update on container support on Google Cloud Platform
by jbeda on Tuesday June 10, 2014 @01:30PM
(#7873992)
We are particularly excited about Kubernetes. We are taking the ideas for how we do cluster management at Google and creating an open source project to manage Docker containers.
Re:An update on container support on Google Cloud Platform
by nivertech on Tuesday June 10, 2014 @07:21PM
(#7875863)
Note that some Google's Debian Wheezy images are specifically optimized for their Andromeda SDN [1], especially when they running on us-central1-b or europe-west1-a zones, where Andromeda is enabled [2].
Also Ubuntu 14.04 LTS has much worse performance than Ubuntu 12.04 LTS, so it's better to stick to 12.04 LTS for now.
In my opinion it's better to port to Debian or to run Ubuntu image inside container hosted on Debian, until they will have native Ubuntu support.
Re:An update on container support on Google Cloud Platform
by michaelmior on Tuesday June 10, 2014 @03:36PM
(#7874811)
"Everything at Google, from Search to Gmail, is packaged and run in a Linux container." Was this something which Google had previously disclosed? Seems a bit surprising to me.
Re:An update on container support on Google Cloud Platform
by zeroxfe on Tuesday June 10, 2014 @07:51PM
(#7875953)
The statically linked binaries run inside the containers. Static linking gives you a certain kind of portability (no need for library dependencies on the machine.) The containers give you isolation, resource management, etc.
Re:An update on container support on Google Cloud Platform
by zmanian on Tuesday June 10, 2014 @11:30PM
(#7876400)
The way it has been explained to me is that Google's fat binaries have no dependencies beyond libc. The task scheduler would just have to deploy that one file to one or more machines to get it to run.
Containers are much more flexible that statically linked binaries. You could have multiple binaries in a container sharing a common set of dynamically linked files.
Fat binaries inside containers sounds a bit like the worst of both worlds...
Re:An update on container support on Google Cloud Platform
by menage on Wednesday June 11, 2014 @03:16AM
(#7876801)
If you want unrelated jobs to share library binaries, you need to stick to explicit release schedules for those libraries, which means more coordination between the various app teams and the teams that own the libraries.
When the size of the libraries (megabytes) is compared with the typical heap size of running jobs (gigabytes - a small number of large instances per job is typically a lot more efficient than a large number of small instances) the space savings of shared libraries become pretty negligible.
Back then, one of the main bottlenecks in the system was the central scheduler, in particular the amount of work it had to do tracking what binary packages were installed on each machine and what were needed for the candidate jobs it might plan to run on those machines. Having many packages per job just makes the scheduler bottleneck worse.
There were two places where it did actually make sense to share packages between jobs:
- Java Virtual Machine and supporting libraries
- libc
These were external (so changed much less frequently), quite large, and were needed by very large numbers of jobs, so the space savings of having one copy of each needed version per machine outweighed the extra scheduling load required for them.
Re:An update on container support on Google Cloud Platform
by thockingoog on Wednesday June 11, 2014 @01:35AM
(#7876616)
Static binaries are congruent to containers, in a way. Both bundle an app and dependencies together. There are advantages and disadvantages to both approaches, but Docker-style containers are more elegant in some ways.
Re:An update on container support on Google Cloud Platform
by menage on Wednesday June 11, 2014 @03:21AM
(#7876809)
Google had been using primitive kernel containers (based on cpusets and fake NUMA emulation) since early 2007 - this was quite some time prior to getting cgroups into the mainline kernel.
Re:An update on container support on Google Cloud Platform
by SEJeff on Tuesday June 10, 2014 @08:11PM
(#7876009)
I'm guessing you weren't aware that google is the company that wrote the overwhelming majority of the cgroup subsystem and much of the namespace bits that lxc/docker use.
Paul Menage and Rohit something were the two biggest kernel guys on it if memory serves. I used to read the LKML firehose actively and gave up eventually.
Re:An update on container support on Google Cloud Platform
by menage on Wednesday June 11, 2014 @03:09AM
(#7876784)
Yes, we contributed the core cgroups system, and a good chunk of the memory and I/O cgroup subsystems. Google didn't have much need for namespacing though - when you have complete control over all the code that's running, it's possible to have the common libraries co-ordinating with the cluster management system to assign things like network ports and userids, so there's no need to virtualize IP addresses or userids. Google's containers are (or at least were - I left a few years ago) just for resource isolation.
Re:An update on container support on Google Cloud Platform
by thockingoog on Tuesday June 10, 2014 @04:36PM
(#7875160)
We've been talking about containers for years. I presented some of our problems with them back in 2011, and I know we've been talking about it before that. :)
Re:An update on container support on Google Cloud Platform
by PanMan on Tuesday June 10, 2014 @04:18PM
(#7875045)
I know Google runs at a huge scale, but isn't even for them 2 billion containers / week a LOT? I assume a lot of these only run for a really short time? Are containers the new scripts?
Re:An update on container support on Google Cloud Platform
by jbeda on Tuesday June 10, 2014 @04:23PM
(#7875082)
I can't give specifics but a lot of these are short lived. For example, if you launch a MapReduce it'll typically launch containers for each of the workers and then take them down when the MR is done.
This also doesn't speak to the number of long running containers. There are plenty that don't stop/start during the week I grabbed that number.
Re:An update on container support on Google Cloud Platform
by derefr on Tuesday June 10, 2014 @04:24PM
(#7875083)
If docker images are fancy static binaries, then docker containers are fancy OS processes. Going through two billion OS PIDs in a week doesn't seem that hard.
Re:An update on container support on Google Cloud Platform
by cmelbye on Wednesday June 11, 2014 @03:24AM
(#7876822)
> Are containers the new scripts?
They certainly seem to work well for that. Heroku, for example, uses containers for not just persistent processes (application servers, workers) but also short-lived processes. Tasks that run on a schedule (hourly, daily, etc.) are run by, you guessed it, starting up a container running a processes which exits when it's finished. One-off commands like maintenance scripts or REPLs work the same way.
https://github.com/GoogleCloudPlatform/kubernetes
Also Ubuntu 14.04 LTS has much worse performance than Ubuntu 12.04 LTS, so it's better to stick to 12.04 LTS for now.
In my opinion it's better to port to Debian or to run Ubuntu image inside container hosted on Debian, until they will have native Ubuntu support.
[1] http://gigaom.com/2014/04/02/google-launches-andromeda-a-sof...
[2] http://googlecloudplatform.blogspot.co.il/2014/04/enter-andr...
Citation needed.
The simpler solution is to use their ready-made Debian images, which can be easily configured via CLI or API.
My slides from the talk: https://speakerdeck.com/jbeda/containers-at-scale PDF: http://slides.eightypercent.net/GlueCon%202014%20-%20Contain...
Did Google switch to containers in a year? Maybe that answer is in your slides? If so crazy...
Containers are much more flexible that statically linked binaries. You could have multiple binaries in a container sharing a common set of dynamically linked files.
Fat binaries inside containers sounds a bit like the worst of both worlds...
When the size of the libraries (megabytes) is compared with the typical heap size of running jobs (gigabytes - a small number of large instances per job is typically a lot more efficient than a large number of small instances) the space savings of shared libraries become pretty negligible.
Back then, one of the main bottlenecks in the system was the central scheduler, in particular the amount of work it had to do tracking what binary packages were installed on each machine and what were needed for the candidate jobs it might plan to run on those machines. Having many packages per job just makes the scheduler bottleneck worse.
There were two places where it did actually make sense to share packages between jobs:
- Java Virtual Machine and supporting libraries - libc
These were external (so changed much less frequently), quite large, and were needed by very large numbers of jobs, so the space savings of having one copy of each needed version per machine outweighed the extra scheduling load required for them.
Essentially it comes across as "I'm a fan of your early work, but nothing you've done since matters."
This also doesn't speak to the number of long running containers. There are plenty that don't stop/start during the week I grabbed that number.
[1] http://www.morganclaypool.com/doi/abs/10.2200/S00516ED2V01Y2...
They certainly seem to work well for that. Heroku, for example, uses containers for not just persistent processes (application servers, workers) but also short-lived processes. Tasks that run on a schedule (hourly, daily, etc.) are run by, you guessed it, starting up a container running a processes which exits when it's finished. One-off commands like maintenance scripts or REPLs work the same way.