Re:Large-scale cluster management at Google with Borg
by gtaylor on Thursday May 07, 2015 @12:36PM
(#9506335)
I've been hesitant to spend a ton of time playing with Kubernetes yet, since it is still in such a state of flux (and the docs are woefully inadequate). With that said, if anyone can make something like this awesome, I'd put my money on Google.
Re:Large-scale cluster management at Google with Borg
by WestCoastJustin on Thursday May 07, 2015 @02:59PM
(#9507427)
They host pretty much everything -- Maps, Gmail, Search, etc. Highly suggest reading the Wired link below as it talks about this in detail [1], but if you want a more technical look, check out the Youtube video [2], or the Borg paper [3]. One really interesting development, is that Mesos + Marathon + Chronos + Docker, puts a very similar system into the hands of your average IT team today [4].
Re:Large-scale cluster management at Google with Borg
by dekhn on Friday May 08, 2015 @09:31AM
(#9511582)
When I ran Exacycle http://googleresearch.blogspot.com/2013/12/groundbreaking-si... the worker pool was just a low-priority (priority 0 in the paper) job like everything else, soaking up idle cycles. The command and control jobs were just regular borg jobs that managed the worker pools by assigning tasks to them.
So whatever else was going on, protein folding and drug docking and telescope simulation was soaking up whatever leftover cycles there were.
Re:Large-scale cluster management at Google with Borg
by _dark_matter_ on Thursday May 07, 2015 @01:00PM
(#9506529)
Can anyone explain how MapReduce is built on top of Borg, when Borg doesn't support scheduling with data locality? I'm having a hard time reconciling that. One of the principal use cases of the MapReduce framework is reducing data movement...
Re:Large-scale cluster management at Google with Borg
by nrr on Thursday May 07, 2015 @01:14PM
(#9506647)
From the MapReduce paper: "The MapReduce master takes the location information of the input files into account and attempts to schedule a map task on a machine that contains a replica of the corresponding input data."
One of two possible designs immediately come to mind: (1) The MapReduce master interacts with Borg directly on behalf of the programmer and places a requirement on the task to run on or near any node that has a replica of the data required or (2) for each Borglet, there is an accompanying MapReduce agent running to accept RPCs from the MapReduce master.
Re:Large-scale cluster management at Google with Borg
by thrownaway2424 on Thursday May 07, 2015 @01:41PM
(#9506856)
You seem to be suggesting that using local i/o to read inputs would be preferable to using remote i/o, but I don't think that's a supportable statement in a datacenter with a high-performance network.
Re:Large-scale cluster management at Google with Borg
by nrr on Thursday May 07, 2015 @04:29PM
(#9508058)
Even with high-performance networks in place, you absolutely must budget your bandwidth between management communications, low-latency service communications, and batch job communications, amongst possibly even others. Having data locality is still a really, really big priority.
(Furthermore, building out separate networks is extremely cost-prohibitive once you've expanded out to a total footprint in the low-1e6 ballpark.)
Think about it this way: In recent history, we've added L3 cache to commodity processors because keeping data on-chip is still faster, in 2015, than going off-chip to RAM. The same analog holds when trying to fetch from local disk versus remote disk.
Brendan Gregg actually provides a really good time-stretched view of various operations in his book on performance tuning. Go give that book a read; there's some _really_ good stuff there.
Re:Large-scale cluster management at Google with Borg
by thrownaway2424 on Friday May 08, 2015 @01:20AM
(#9510094)
It just doesn't seem very likely to me. It seems a lot more likely that the data would be spread around on thousands of individual hosts, one megabyte at a time. The only processes by which I imagine the entire contents of one shard of a mapreduce ending up in memory of a single host are 1) a miracle, 2) the previous stage of the mapreduce wrote it there, in which case it is effectively remote access before the fact, and 3) the data lives there all the time for some other reason, in which case you don't have a mapreduce, you have a coprocessor.
Re:Large-scale cluster management at Google with Borg
by VikingCoder on Tuesday May 12, 2015 @10:52AM
(#9532170)
I agree about the one megabyte at a time. I disagree that that implies it's likely evicted from RAM.
For one, if I'm running a MapReduce, it's either likely on the same data someone else is investigating... Or I'm likely to have to re-run my MapReduce a few times, calculating exactly what I'm looking for.
So, I may be cheating by saying "Only the first time is the data cold. Every other time, the data is likely to be hot in RAM."
Because you'd be right to say, "Well, duh. I'm talking about the FIRST time. And since I'm talking about the first time, moving the data across the network wouldn't be that bad. After that, yes, of course it will still likely be hot, just as you describe."
My only real response is, I think it depends on how much data, versus how much code you're talking about. And can you maybe even broadcast the code to the right nodes, to even more effectively use the network.
"in which case you don't have a mapreduce, you have a coprocessor."
...a coprocessor which uses the MapReduce API. Yes, I think that is exactly what you have. And it works whether the data is local or remote, cold or hot. One unified API for all of those things, and it's probably optimized to work local and hot, because I'm betting more than half of all MapReduces actually work out to be local and hot.
Re:Large-scale cluster management at Google with Borg
by powera on Thursday May 07, 2015 @01:34PM
(#9506799)
The MR master handles it.
Mapreduce workers can request shards (work) that have data local to the host it is running on. For large mapreduces this basically works because both the data and the tasks are in most of the cluster, and for small mapreduces it doesn't matter.
Re:Large-scale cluster management at Google with Borg
by wffurr on Thursday May 07, 2015 @01:34PM
(#9506800)
Borg also uses cgroups, just like LXC.
From the paper: "We use a Linux chroot jail as the primary security isolation mechanism between multiple tasks on the same machine… all Borg tasks run inside a Linux cgroup-based resource container."
Re:Large-scale cluster management at Google with Borg
by samkone on Thursday May 07, 2015 @02:01PM
(#9507000)
Remember that Borg is an old thing. At the time they probably started out with chroot jails, because cgroups weren't ready yet. But Kubernetes, Mesos use docker and cgroups(Mesos).
Re:Large-scale cluster management at Google with Borg
by zobzu on Thursday May 07, 2015 @01:10PM
(#9506613)
Not really- chroot provides no isolation, namespaces do.
choot is the visible portion (changes root.. ie changes /mnt/blah/bleh to /) but not "the foundation"
Things can in fact work isolated without chroot, too. Its just convenient.
Re:Large-scale cluster management at Google with Borg
by exacube on Thursday May 07, 2015 @05:22PM
(#9508391)
Note that chroot will provide containerization, but without isolation (processes can always fork out of their chroot jails). You still need things like pid namespaces etc.
Re:Large-scale cluster management at Google with Borg
by AdmiralAsshat on Thursday May 07, 2015 @01:00PM
(#9506528)
Does anyone else feel like calling your massive infrastructure software "Borg" might be tempting fate? I mean, I'm not superstitious, but I would be weary of calling my satellite network "Skynet."
Exciting times!
[1] http://www.wired.com/2013/03/google-borg-twitter-mesos/
[2] https://www.youtube.com/watch?v=0ZFMlO98Jkc
[3] https://research.google.com/pubs/archive/43438.pdf
[4] https://www.youtube.com/watch?v=hZNGST2vIds
- websearch
- google brain
- video transcoding for youtube
- appengine (think snapchat)
- gmail
Almost everything Google does runs on Borg.
So whatever else was going on, protein folding and drug docking and telescope simulation was soaking up whatever leftover cycles there were.
One of two possible designs immediately come to mind: (1) The MapReduce master interacts with Borg directly on behalf of the programmer and places a requirement on the task to run on or near any node that has a replica of the data required or (2) for each Borglet, there is an accompanying MapReduce agent running to accept RPCs from the MapReduce master.
(Furthermore, building out separate networks is extremely cost-prohibitive once you've expanded out to a total footprint in the low-1e6 ballpark.)
Think about it this way: In recent history, we've added L3 cache to commodity processors because keeping data on-chip is still faster, in 2015, than going off-chip to RAM. The same analog holds when trying to fetch from local disk versus remote disk.
Brendan Gregg actually provides a really good time-stretched view of various operations in his book on performance tuning. Go give that book a read; there's some _really_ good stuff there.
No, seriously.
Picture if all of your data is in RAM, somewhere. If you can move your program to that machine quickly, then that's a huge win.
For one, if I'm running a MapReduce, it's either likely on the same data someone else is investigating... Or I'm likely to have to re-run my MapReduce a few times, calculating exactly what I'm looking for.
So, I may be cheating by saying "Only the first time is the data cold. Every other time, the data is likely to be hot in RAM."
Because you'd be right to say, "Well, duh. I'm talking about the FIRST time. And since I'm talking about the first time, moving the data across the network wouldn't be that bad. After that, yes, of course it will still likely be hot, just as you describe."
My only real response is, I think it depends on how much data, versus how much code you're talking about. And can you maybe even broadcast the code to the right nodes, to even more effectively use the network.
"in which case you don't have a mapreduce, you have a coprocessor."
...a coprocessor which uses the MapReduce API. Yes, I think that is exactly what you have. And it works whether the data is local or remote, cold or hot. One unified API for all of those things, and it's probably optimized to work local and hot, because I'm betting more than half of all MapReduces actually work out to be local and hot.
Mapreduce workers can request shards (work) that have data local to the host it is running on. For large mapreduces this basically works because both the data and the tasks are in most of the cluster, and for small mapreduces it doesn't matter.
From the paper: "We use a Linux chroot jail as the primary security isolation mechanism between multiple tasks on the same machine… all Borg tasks run inside a Linux cgroup-based resource container."
I think was in RHEL 6.2 possibly even 6.0
You may run O(10) containers on a single machine. When you run O(10k)+ containers in a cluster you need new tools to manage things.
Things can in fact work isolated without chroot, too. Its just convenient.
https://en.m.wikipedia.org/wiki/Skynet_(satellites)