> Most of the world manages to run their binaries on a mainline kernel

Most of the world doesn’t need to worry about their kernel - this is a good thing. But at FAANG scale, you inevitably need to make changes and optimizations.

Eg: “Twitter has a kernel team!?” - https://danluu.com/in-house/

I thought the kernel was pretty stable. My first team at twitter I think we ran into 2 or 3 kernel bugs that year. I couldn't believe it but for some problems the blaming the kernel is a real debugging step (and even the bios lol). I think I had two perf problems turn into upstream patches thanks to the kernel team.

It's not only about bugs, but if you know your workload and your systems you can do some tricks a general purpose kernel can't do to tweak it for your needs and get a benefit here or there for your code.

that was a very interesting article; thanks for posting it :)

Np, I saw it on HN last week :)

Discussion: https://news.ycombinator.com/item?id=28691676

I remember reading an article about new system calls proposed by Google. It was a scheduler yield system call that allowed the current task to yield its remaining processing time to another task. It would presumably be used to make goroutines more performant.

Not sure if the system call was accepted or if it still exists as Google specific kernel code. I can't seem to find the original article either...

Here: https://youtu.be/KXuZi9aeGTw

Jump to the 15-minute mark for the three new system calls, switchto_wait, switchto_resume, switchto_switch.

I think you might be thinking of https://lwn.net/Articles/826860/

As an example, cgroup was initially developed in Google: https://en.wikipedia.org/wiki/Cgroups

Cgroups is a great example of the problems faced here.

The cgroups code/API that made it into the upstream kernel was the product of a couple of years of internal experimentation at Google into ways to do kernel-level resource isolation, and was pretty different from the approach that was used internally in production on a rather older kernel version. (A bit like Borg vs Kubernetes). And it still took a couple more years for cgroups to replace the original internal mechanisms, since the internal way worked OK and upgrading the kernel across so many machines was risky.

And now we have CgroupsV2 to fix all of the issues with the original

Yes, and it shows (it's notoriously poorly designed)

But it has the best design property of software: it exists and works in production.

That is what I tell myself when I get to review offshoring code.

Is Google notorious from poor design, or is there some other reason why cgroup being poorly designed would have to do it being designed by Google?

When it comes to Linux kernel work, yes.

From Android point of view, poor design.

Arguably, every line of code in the Linux kernel came from someone who couldn't run their binaries on mainline.

It is sometimes hard to imagine the scale of Google. If you can implement a task-switching mechanism with lower overhead and save 0.1% of CPU cycles across the fleet you can save many millions of dollars a year. So they have their own thread-switching mechanism which their binaries that can't run without. If you can improve process isolation in a way that allows you to pack 10% more tasks (by RAM or CPU) onto a single machine without affecting performance significantly you have just saved billions. So they have different isolation interfaces that Borg can't run without.

Both of these are completely feasible savings and make custom binaries and a custom kernel completely with it.

Ask a Googler about the size of their environment variables.

ever wonder why linux supports command line arguments that are 2mb long?

Ever wondered absl flags library has a --flagfile special flag?

https://abseil.io/docs/python/guides/flags#special-flags

I wonder if ABSL supports gzipped flags...

Probably not, since --flagfile accepts a list of filenames :-)

https://github.com/abseil/abseil-cpp/blob/1ae9b71c474628d60e...

Does Linux need explicit support for this? Command line arguments and environment variables are copied to the stack of the new process. I would expect the stack's size to be the limiting factor.

Before they relaxed it 15 years ago, Linux had a hard limit at 128KiB for args + env on x86. After the change it was variable up to 2MiB.

https://github.com/torvalds/linux/commit/b6a2fea39318e43fee8...

I'm afraid of what they're likely to show me...

Your fears are completely founded.

In addition to hardware support, the article also mentioned adjustments to the perennial Linux pain in the asses that are the scheduler and the OOM killer.

Not sure - are Google running their servers with custom-made processors as well?

A lot of it is just mundane details. Some of it leaks out in ABSL, for example some boring trivia is at:

https://github.com/abseil/abseil-cpp/blob/master/absl/base/i...

They also have for years had major patches to TCP which Linux refuses to adopt. Some details at:

https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tc...

These won't make binaries "not run" as such but they are necessary for the correct and efficient operation of large-scale distributed systems.

QUIC, Snap/Pony, and user-space thread scheduling are strictly superior to what happens inside the kernel and their development makes the delta between prodkernel and upstream kernel less relevant over time. They also, collectively, make Linux itself less relevant.

Linux kernel is the most sucessful free beer UNIX clone, but it is by no means the only one, and if the trends on IoT and cloud infrastructure are any measure, in a couple of decades we might still live under POSIX monoculture to some extent, but Linux most likely won't be the kernel underneath.

Not CPUs but a lot of other components are completely custom, developed in-house. A well known example: TPUs. Google is one of the world's largest manufacturer of servers and other DC equipment, actually, it's just that none of that is for sale (though a lot is for rent :) ).

Unless they've been building any factories I don't know about, their contractors are the manufacturers. I'm talking about companies based in Taiwan like Quanta, Compal, Clevo, and others. If you've ever been to a Computex Taipei you have seen the booths of the actual manufacturers of Google's hardware.

https://en.wikipedia.org/wiki/Computex

This is a meaningless nitpicking. It is like nitpicking Apple is not a manufacturer of hardware since the actual assembly happens under some contractor. For all intent and purpose, and even technically, Google is the largest manufacturer of data center-sized supercomputers.

Like when that Pluto Switch got leaked back in 2012

https://www.wired.com/2012/09/pluto-switch/

See this for an early example: https://news.ycombinator.com/item?id=542531

That's covered earlier (full quote):

> Those patches implement various internal APIs (e.g. for Google Fibers), provide hardware support, add performance optimizations, and contain other "tweaks that are needed to run binaries that we use at Google".

I'd assume if you're making allowances for custom-made processors at the kernel level you wouldn't also want that leaking into your user land binaries as well.

Why not? They could have kernel accommodation for one feature, and user-land leakage for another.

Also, for some features you need both kernel and userland cooperation. Just think of eg fuse or io_uring or mmap that are in the public kernel. You can surely imagine that Google might brew up similar features.

(I used to work at Google. But I didn't have any special insider knowledge about their kernel stuff. Which is good, so I can't violate any lingering NDAs here with my speculation..)

Kind of, I've seen some non-public Intel SKUs in the fleet but afaik they just had tweaked core counts and clocks for better perf/watt. They were also exploring non-intel architectures at some point, ppc especially but no significant prod deployment yet. And then I'm sure you've heard of special purpose hardware like TPUs

Some of them, yes! At least for machine learning. They also have custom silicon in their phones (and the phones run a Linux).