Large-scale cluster management at Google with Borg

Re:Large-scale cluster management at Google with Borg by mandeepj on Thursday May 07, 2015 @02:14PM (#9507093)

Can anyone throw some light on what google is doing with tens of thousands of machines in these clusters? Are they hosting crawlers on them?

[ Parent | on HN ]

Re:Large-scale cluster management at Google with Borg by WestCoastJustin on Thursday May 07, 2015 @02:59PM (#9507427)

They host pretty much everything -- Maps, Gmail, Search, etc. Highly suggest reading the Wired link below as it talks about this in detail [1], but if you want a more technical look, check out the Youtube video [2], or the Borg paper [3]. One really interesting development, is that Mesos + Marathon + Chronos + Docker, puts a very similar system into the hands of your average IT team today [4].

[1] http://www.wired.com/2013/03/google-borg-twitter-mesos/

[2] https://www.youtube.com/watch?v=0ZFMlO98Jkc

[3] https://research.google.com/pubs/archive/43438.pdf

[4] https://www.youtube.com/watch?v=hZNGST2vIds

[ Parent | on HN ]

Re:Large-scale cluster management at Google with Borg by coops on Thursday May 07, 2015 @02:50PM (#9507359)

- crawlers

- websearch

- google brain

- video transcoding for youtube

- appengine (think snapchat)

- gmail

Almost everything Google does runs on Borg.

[ Parent | on HN ]

Re:Large-scale cluster management at Google with Borg by dekhn on Friday May 08, 2015 @09:31AM (#9511582)

When I ran Exacycle http://googleresearch.blogspot.com/2013/12/groundbreaking-si... the worker pool was just a low-priority (priority 0 in the paper) job like everything else, soaking up idle cycles. The command and control jobs were just regular borg jobs that managed the worker pools by assigning tasks to them.

So whatever else was going on, protein folding and drug docking and telescope simulation was soaking up whatever leftover cycles there were.

[ Parent | on HN ]