Re:Borg: The Next Generation [pdf]
by remus on Friday May 01, 2020 @02:13AM
(#23039526)
The resource usage section at the end was really interesting, and surprising to me. 1% of jobs use 99% of resources! It would be interesting to try and understand how this pattern came about and if there's particular engineering decisions that tend to lead to this situation where you have a handful of incredibly resource intensive jobs and loads of very lightweight jobs.
[ Parent
| on HN ]
That seems more related to weird choice of denominator than to any underlying facts of large scale cluster management. Large services have stable names and run forever. Search is just "search" and bigtable is just "bigtable" (some irrelevant details have been elided for clarity). If I run a batch logs analysis job though it is transient and has a unique name every time I run it. So there's a huge long tail of transient job _names_ and a few dominant services with permanent names, which makes the number-of-jobs denominator quirky.