News for hackers. Stuff that matters.
Re:BPF at Facebook and beyond by saagarjha on Thursday October 10, 2019 @04:45PM (#21218980)
> Facebook, he began, has an upstream-first philosophy, taken to an extreme; the company tries not to carry any out-of-tree patches at all. All work done at Facebook is meant to go upstream as soon as it practically can. The company also runs recent kernels, upgrading whenever possible.

I was chatting to a Facebook engineer on their use of BPF this summer and heard the same thing, which was surprising to me. There seem to be a number of companies that take advantage of Linux being licensed under GPL and keep their own forks/patches of the kernel that they use internally (anecdotally, I’ve heard Google does this), and the they stay on some old version, which apparently Facebook doesn’t do.

Re:BPF at Facebook and beyond by marios on Thursday October 10, 2019 @05:20PM (#21219303)
Upstreaming means the 'cost of maintainance' is shared. If you have out of tree patches, and upstream changes the API, either you have to update your patches or you have to keep running an older version. Possibly missing out on other improvements.

Getting things upstream can be more time consuming, but it's a worthwhile investment IMO.

Re:BPF at Facebook and beyond by saagarjha on Thursday October 10, 2019 @05:49PM (#21219523)
Right. A lot of companies seem to do this calculus and come up with “my proprietary add ons are worth more than upstreaming this and having to update the patches whenever we want to upgrade”, unfortunately :(
Re:BPF at Facebook and beyond by AndyMcConachie on Thursday October 10, 2019 @06:01PM (#21219631)
Either that or the code you developed for the Linux kernel is so special no one else wants it and you can't get it merged.

I've seen this happen with Linux kernel stuff as well as other OSS projects.

Re:BPF at Facebook and beyond by darksaints on Thursday October 10, 2019 @06:43PM (#21219929)
> There seem to be a number of companies that take advantage of Linux being licensed under GPL and keep their own forks/patches of the kernel that they use internally (anecdotally, I’ve heard Google does this), and the they stay on some old version

:::cough::: Amazon :::cough:::

Re:BPF at Facebook and beyond by dwoozle on Thursday October 10, 2019 @06:53PM (#21220008)
Well Facebook is just a user of compute, not a seller of it. Google and Amazon sell compute as a business. The incentives are different. Facebook, despite its size, will never be a large fraction of Linux usage, so they are much wiser to benefit from the improvements of the community and give and take from the master. Amazon sells compute so their contributions would be shared by only the three major cloud vendors.
Re:BPF at Facebook and beyond by habitue on Thursday October 10, 2019 @05:35PM (#21219419)
I can't wait until this is the norm. Facebook (and presumably, Google if they wanted to) can do this with an army of engineers and some excellent continuous deployment systems. I'm hoping these kinds of systems become commoditized over time and regular companies can stay close to the latest kernel version at all times
Re:BPF at Facebook and beyond by Twirrim on Thursday October 10, 2019 @06:47PM (#21219956)
Take a look at the list of major contributors to the kernel, just covering top 50 between v5.0 and v5.3:

   $ git log v5.3...v5.0 | grep "^Author:" | cut -d"@" -f2 | sed 's/>//' | sort | uniq -c | sort -n | tail -50
        167 samsung.com
        188 sang-engineering.com
        190 acm.org
        192 broadcom.com
        196 collabora.com
        206 ingics.com
        207 infradead.org
        207 lixom.net
        219 pengutronix.de
        223 glider.be
        232 c-s.fr
        232 microchip.com
        247 mediatek.com
        251 roeck-us.net
        329 st.com
        341 fb.com
        341 socionext.com
        346 netronome.com
        360 ti.com
        380 canonical.com
        384 linuxfoundation.org
        387 nvidia.com
        388 codeaurora.org
        398 arndb.de
        424 embeddedor.com
        460 renesas.com
        469 chris-wilson.co.uk
        498 baylibre.com
        504 suse.com
        516 chromium.org
        539 suse.de
        540 lst.de
        554 oracle.com
        652 nxp.com
        682 bootlin.com
        690 davemloft.net
        697 linutronix.de
        774 arm.com
        781 linux.ibm.com
        948 google.com
       1052 linaro.org
       1230 huawei.com
       1315 mellanox.com
       1333 linux-foundation.org
       1477 linux.intel.com
       1501 kernel.org
       1851 amd.com
       2229 redhat.com
       2548 intel.com
       4373 gmail.com
There's the usual suspects in there (after a quick double check, it turns out microsoft.com just missed the cut-off with 140 commits, plus 2 from linux.microsoft.com, Amazon has authored 71, and alibaba have 139)

No one wants to maintain out-of-tree patches, it's a complete pain. Google was doing it extensively with the Android project for the longest time, but they've been working hard at getting those all in to upstream to drastically reduce the work involved in updating the Android kernel.

Re:BPF at Facebook and beyond by josefbacik on Thursday October 10, 2019 @06:55PM (#21220018)
Keep in mind most of us don’t use our @fb.com addresses. LWN keeps track of companies by keeping track of who works where so our numbers are much higher than this list shows.
Re:BPF at Facebook and beyond by Twirrim on Thursday October 10, 2019 @07:04PM (#21220091)
I believe the same applies to Amazon to an extent, too.
Re:BPF at Facebook and beyond by ajdlinux on Thursday October 10, 2019 @07:50PM (#21220355)
Applies across a bunch of companies (I use my IBM email but I know quite a few kernel devs here who prefer to keep their open source contributions under their personal email).
Re:BPF at Facebook and beyond by dmix on Thursday October 10, 2019 @08:00PM (#21220404)
A lot of them have it tied to PGP and other stuff so it makes sense. Identity matters in OSS.
Re:BPF at Facebook and beyond by jumpingmice on Thursday October 10, 2019 @05:57PM (#21219597)
Google can't necessarily upstream everything because of social problems in the kernel process. For example their datacenter TCP improvements have never been accepted by the gatekeeper of the net subsystem, which was a significant motivation to develop QUIC.
Re:BPF at Facebook and beyond by fdee on Friday October 11, 2019 @07:41AM (#21223357)
I'm not sure where you have heard about this? Their DCTCP extensions have never been posted in the first place to a public list as of today. Pretty much all of the core TCP developers for the (upstream) kernel's networking subsystem are employed by Google and doing an excellent job. That said, I would love to see their extensions integrated into the upstream tcp_dctcp module.
Re:BPF at Facebook and beyond by fdee on Friday October 11, 2019 @08:04AM (#21223498)
Is Facebook running DCTCP in production these days?
Re:BPF at Facebook and beyond by the8472 on Thursday October 10, 2019 @06:23PM (#21219804)
They did get BBR into the kernel though, and many moons ago BQL too which was a prerequisite.
Re:BPF at Facebook and beyond by jabl on Friday October 11, 2019 @01:18AM (#21221749)
Isn't DCTCP generalized by TCP Prague and L4S? Which, if they get the IETF stamp of approval and the potential patent issues around L4S get sorted out, I'd guess would be implemented in the upstream Linux kernel pretty quickly.
Re:BPF at Facebook and beyond by pnako on Thursday October 10, 2019 @07:41PM (#21220312)
Social problems, a.k.a. Linux must work for everyone and not just Google.
Re:BPF at Facebook and beyond by bifrost on Thursday October 10, 2019 @06:07PM (#21219671)
Reinventing TCP over UDP is sortof silly, I hope they have a better reason than "they don't want to upstream our changes" lol.
Re:BPF at Facebook and beyond by wmf on Thursday October 10, 2019 @06:36PM (#21219885)
I think the inability to upstream changes into Windows and (ironically) old versions of Android are bigger motivations for using UDP.
Re:BPF at Facebook and beyond by jumpingmice on Thursday October 10, 2019 @06:16PM (#21219740)
Isn't it a pretty good reason? gRPC is terrible in a datacenter context without Google's internal TCP fixes that Linux won't adopt (and which have been advocated for in numerous conference papers since at least 2009). If they are steadfast cavemen what other workaround exists?
Re:BPF at Facebook and beyond by ToFab123 on Friday October 11, 2019 @10:08AM (#21224493)
Apparently Microsoft is considering gRPC as an future replacement for WCF, so that might change. https://news.ycombinator.com/item?id=21055487
Re:BPF at Facebook and beyond by greglindahl on Friday October 11, 2019 @12:36AM (#21221599)
The standard workaround is to send short messages using UDP and long ones using TCP.
Re:BPF at Facebook and beyond by derefr on Thursday October 10, 2019 @06:55PM (#21220015)
What parts of gRPC are fixed by using it over QUIC vs. TCP (presuming intra-DC traffic and equally long-lived flows)?
Re:BPF at Facebook and beyond by jumpingmice on Thursday October 10, 2019 @09:02PM (#21220656)
Latency caused by packet loss. TCP needs microsecond timestamps and the ability to tune RTOmin down to 1ms before it is suitable for use in a datacenter. With the mainline kernel TCP stack you are looking at, at a minimum, a 20ms penalty whenever a packet is dropped.
Re:BPF at Facebook and beyond by toast0 on Friday October 11, 2019 @01:57AM (#21221891)
TCP over UDP seems rather silly to me, but congestion control and segmentation in userland is pretty useful. Especially so, since Google and partners have made an ecosystem where kernel updates on deployed devices don't happen.