News for hackers. Stuff that matters.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by taldo on Tuesday November 02, 2021 @12:51PM (#29083858)
Ah, the laughs (Xoogler since 2020). It was a lot easier, at least last year: you'd use "flex" quota from your PA pool (product area) for Spanner and Borg, write some code for your server, a few configs here and there, and you'd be ready and serving.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by bhickey on Tuesday November 02, 2021 @06:58PM (#29088653)
About six years ago I had a resource manager deny me a database instance the very same day it became available for flex in another product area. I tried to "Hey Mister" resources from someone in that group to no avail. Eventually I wrote a high-durability key-value store on top of our source control system and told them they could give me my database or I'd be deploying to prod.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by plaguuuuuu on Wednesday November 03, 2021 @09:46AM (#29094474)
>Eventually I wrote a high-durability key-value store on top of our source control system and told them they could give me my database or I'd be deploying to prod.

That's diabolical. I knew Google should never have dropped the "do not be evil" thing

Re:Google's infamous internal 2010 “I just want to serve 5TB... by dilyevsky on Tuesday November 02, 2021 @03:20PM (#29086172)
That video came out a few years before flex appeared I think at a time they were having a sort of “resource crunch” on the heels of growth spur following the GFC.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by the-rc on Tuesday November 02, 2021 @05:37PM (#29087729)
It wasn't that there was a crunch — that had always existed. There just wasn't all the tooling to implement anything like flex. At least this video was made after "buying Borg quota" was a normal thing. Before it, you had to "buy" regular machines and donate/assimilate them into Borg. Then after X days you'd receive your quota, minus a Borg "tax" of 10% to cover borglet and system daemons' overhead.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by menage on Tuesday November 02, 2021 @09:16PM (#29089944)
And then they got Steamrolled away if you didn't use them enough.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by the-rc on Tuesday November 02, 2021 @11:31PM (#29090824)
Hah, I think that at least at the beginning, the tools wanted to steamroll the Bigtable service, too. As the one that had to vet the changes every week, I had to go explain that it wasn't really neither our quota nor our usage. For Colossus, sometimes large new clusters would go from very idle to super busy and under provisioned within days or few weeks, AFTER the steamrolling was already done, just because large users moved in. We then had to bring up new curators (masters) quickly, many times with quota that didn't exist. Quite often, when out of options, I ended up taking precious p360 quota from the storage monitoring user to mint production resources for Colossus. Fun times.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by exikyut on Wednesday November 03, 2021 @01:58AM (#29091506)
So before Borgmon existed, the quota unit was basically an entire machine? No virtualization?

Oh wait, this was probably 2008, VMWare had only just figured out JITed software virtualization a couple years prior. Makes a bit more sense now. And now the containerization thing makes a tad more sense: Google basically skipped over the "use QEMU" (or more recently, Firecracker) phase everyone else is now going through.

Re:Google's infamous internal 2010 “I just want to serve 5TB... by the-rc on Wednesday November 03, 2021 @10:14AM (#29094866)
Borgmon and Borg are different things, but yes, before the latter, machines were owned by teams. Brian Grant talks a bit about the predecessors, Babysitter and GWQ: https://softwareengineeringdaily.com/2018/04/27/google-clust...

Borg was started in 2003 and by 2006-2007 was already the default way to run things, even though isolation wasn't perfect then. I think it started with chroot jails, then fake NUMA, then cgroups, which were written for it. It took years before all of web search moved to dedicated Borg machines (their quota belonged to you and nobody else could run on them) and, eventually, shared ones.

Re:Google's infamous internal 2010 “I just want to serve 5TB... by dilyevsky on Wednesday November 03, 2021 @12:12PM (#29096653)
Fun fact - up this thread you replied to one of original cgroups/borg creato’s comment
Re:Google's infamous internal 2010 “I just want to serve 5TB... by rachelbythebay on Wednesday November 03, 2021 @02:51AM (#29091775)
Ah, see, should've given 'em to us! Big rectangular state, you know, #3 machine owner at the time. We didn't charge overhead, probably because it never occurred to us to do it.

People brought machines, we gave 'em quota. Easy enough.

So glad to not be doing that any more.

Re:Google's infamous internal 2010 “I just want to serve 5TB... by the-rc on Wednesday November 03, 2021 @12:29PM (#29096887)
I know the service! The folks in NY had the state flag by their cubicle. My team in 2007-2011 was probably one of the largest users and I think I donated machines to y'all in the old Groningen cluster, before quotas were automated. GFS didn't charge for chunkserver overhead, either, and that mistake took years and lots of pain to fix...
Re:Google's infamous internal 2010 “I just want to serve 5TB... by willidiots on Tuesday November 02, 2021 @04:33PM (#29086982)
Flex was available for certain things (Colossus IIRC, gave you a ton of flex quota) but for others it wasn't. Because This is Google.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by the-rc on Tuesday November 02, 2021 @06:00PM (#29088009)
It was easier to mint and carve out Colossus quota than e.g. Bigtable. I seem to remember that flex for Borg existed, but only in a few locations with enough capacity to back it. You couldn't just retrofit it in clusters where existing, large customers were already granted and using most of the quota.
Re:Google's infamous internal 2010 “I just want to serve 5TB... by dekhn on Tuesday November 02, 2021 @03:42PM (#29086443)
you left out monitoring for reliability which is a major part of this video
Re:Google's infamous internal 2010 “I just want to serve 5TB... by packetslave on Tuesday November 02, 2021 @07:03PM (#29088700)
automon is (was?) a magical thing
Re:Google's infamous internal 2010 “I just want to serve 5TB... by weeks on Wednesday November 03, 2021 @02:07AM (#29091551)
Is. :D