News for hackers. Stuff that matters.
Re:BPF at Facebook and beyond by habitue on Thursday October 10, 2019 @05:35PM (#21219419)
I can't wait until this is the norm. Facebook (and presumably, Google if they wanted to) can do this with an army of engineers and some excellent continuous deployment systems. I'm hoping these kinds of systems become commoditized over time and regular companies can stay close to the latest kernel version at all times
Re:BPF at Facebook and beyond by Twirrim on Thursday October 10, 2019 @06:47PM (#21219956)
Take a look at the list of major contributors to the kernel, just covering top 50 between v5.0 and v5.3:

   $ git log v5.3...v5.0 | grep "^Author:" | cut -d"@" -f2 | sed 's/>//' | sort | uniq -c | sort -n | tail -50
        167 samsung.com
        188 sang-engineering.com
        190 acm.org
        192 broadcom.com
        196 collabora.com
        206 ingics.com
        207 infradead.org
        207 lixom.net
        219 pengutronix.de
        223 glider.be
        232 c-s.fr
        232 microchip.com
        247 mediatek.com
        251 roeck-us.net
        329 st.com
        341 fb.com
        341 socionext.com
        346 netronome.com
        360 ti.com
        380 canonical.com
        384 linuxfoundation.org
        387 nvidia.com
        388 codeaurora.org
        398 arndb.de
        424 embeddedor.com
        460 renesas.com
        469 chris-wilson.co.uk
        498 baylibre.com
        504 suse.com
        516 chromium.org
        539 suse.de
        540 lst.de
        554 oracle.com
        652 nxp.com
        682 bootlin.com
        690 davemloft.net
        697 linutronix.de
        774 arm.com
        781 linux.ibm.com
        948 google.com
       1052 linaro.org
       1230 huawei.com
       1315 mellanox.com
       1333 linux-foundation.org
       1477 linux.intel.com
       1501 kernel.org
       1851 amd.com
       2229 redhat.com
       2548 intel.com
       4373 gmail.com
There's the usual suspects in there (after a quick double check, it turns out microsoft.com just missed the cut-off with 140 commits, plus 2 from linux.microsoft.com, Amazon has authored 71, and alibaba have 139)

No one wants to maintain out-of-tree patches, it's a complete pain. Google was doing it extensively with the Android project for the longest time, but they've been working hard at getting those all in to upstream to drastically reduce the work involved in updating the Android kernel.

Re:BPF at Facebook and beyond by josefbacik on Thursday October 10, 2019 @06:55PM (#21220018)
Keep in mind most of us don’t use our @fb.com addresses. LWN keeps track of companies by keeping track of who works where so our numbers are much higher than this list shows.
Re:BPF at Facebook and beyond by Twirrim on Thursday October 10, 2019 @07:04PM (#21220091)
I believe the same applies to Amazon to an extent, too.
Re:BPF at Facebook and beyond by ajdlinux on Thursday October 10, 2019 @07:50PM (#21220355)
Applies across a bunch of companies (I use my IBM email but I know quite a few kernel devs here who prefer to keep their open source contributions under their personal email).
Re:BPF at Facebook and beyond by dmix on Thursday October 10, 2019 @08:00PM (#21220404)
A lot of them have it tied to PGP and other stuff so it makes sense. Identity matters in OSS.
Re:BPF at Facebook and beyond by jumpingmice on Thursday October 10, 2019 @05:57PM (#21219597)
Google can't necessarily upstream everything because of social problems in the kernel process. For example their datacenter TCP improvements have never been accepted by the gatekeeper of the net subsystem, which was a significant motivation to develop QUIC.
Re:BPF at Facebook and beyond by fdee on Friday October 11, 2019 @07:41AM (#21223357)
I'm not sure where you have heard about this? Their DCTCP extensions have never been posted in the first place to a public list as of today. Pretty much all of the core TCP developers for the (upstream) kernel's networking subsystem are employed by Google and doing an excellent job. That said, I would love to see their extensions integrated into the upstream tcp_dctcp module.
Re:BPF at Facebook and beyond by fdee on Friday October 11, 2019 @08:04AM (#21223498)
Is Facebook running DCTCP in production these days?
Re:BPF at Facebook and beyond by the8472 on Thursday October 10, 2019 @06:23PM (#21219804)
They did get BBR into the kernel though, and many moons ago BQL too which was a prerequisite.
Re:BPF at Facebook and beyond by jabl on Friday October 11, 2019 @01:18AM (#21221749)
Isn't DCTCP generalized by TCP Prague and L4S? Which, if they get the IETF stamp of approval and the potential patent issues around L4S get sorted out, I'd guess would be implemented in the upstream Linux kernel pretty quickly.
Re:BPF at Facebook and beyond by pnako on Thursday October 10, 2019 @07:41PM (#21220312)
Social problems, a.k.a. Linux must work for everyone and not just Google.
Re:BPF at Facebook and beyond by bifrost on Thursday October 10, 2019 @06:07PM (#21219671)
Reinventing TCP over UDP is sortof silly, I hope they have a better reason than "they don't want to upstream our changes" lol.
Re:BPF at Facebook and beyond by wmf on Thursday October 10, 2019 @06:36PM (#21219885)
I think the inability to upstream changes into Windows and (ironically) old versions of Android are bigger motivations for using UDP.
Re:BPF at Facebook and beyond by jumpingmice on Thursday October 10, 2019 @06:16PM (#21219740)
Isn't it a pretty good reason? gRPC is terrible in a datacenter context without Google's internal TCP fixes that Linux won't adopt (and which have been advocated for in numerous conference papers since at least 2009). If they are steadfast cavemen what other workaround exists?
Re:BPF at Facebook and beyond by ToFab123 on Friday October 11, 2019 @10:08AM (#21224493)
Apparently Microsoft is considering gRPC as an future replacement for WCF, so that might change. https://news.ycombinator.com/item?id=21055487
Re:BPF at Facebook and beyond by greglindahl on Friday October 11, 2019 @12:36AM (#21221599)
The standard workaround is to send short messages using UDP and long ones using TCP.
Re:BPF at Facebook and beyond by derefr on Thursday October 10, 2019 @06:55PM (#21220015)
What parts of gRPC are fixed by using it over QUIC vs. TCP (presuming intra-DC traffic and equally long-lived flows)?
Re:BPF at Facebook and beyond by jumpingmice on Thursday October 10, 2019 @09:02PM (#21220656)
Latency caused by packet loss. TCP needs microsecond timestamps and the ability to tune RTOmin down to 1ms before it is suitable for use in a datacenter. With the mainline kernel TCP stack you are looking at, at a minimum, a 20ms penalty whenever a packet is dropped.
Re:BPF at Facebook and beyond by toast0 on Friday October 11, 2019 @01:57AM (#21221891)
TCP over UDP seems rather silly to me, but congestion control and segmentation in userland is pretty useful. Especially so, since Google and partners have made an ecosystem where kernel updates on deployed devices don't happen.