News for hackers. Stuff that matters.
Re:Kubernetes Failure Stories by kortilla on Monday January 21, 2019 @02:22AM (#18958382)
Having a 100gbps nic in a node isn’t the same thing as doing storage at that speed in an HA cluster.

Also, don’t confuse 100 gbe networks where spine links are 100 but the node links are only bonded 10s (much more common at $fang).

Re:Kubernetes Failure Stories by m0zg on Monday January 21, 2019 @04:05AM (#18958896)
Nope. It's all 100GbE throughout as far as I know. And people do work really hard to be able to saturate that bandwidth as it is by no means a trivial task to saturate it through the usual, naive means without the use of RDMA and Verbs. Years ago when I was there it was (IIRC) 40Gbps to each node straight up.

It's a necessity really. All storage at Google has been remote and distributed for at least the past decade. That puts serious demands on network throughput if you want your CPUs to actually do work and not just sit there and wait for data.

Here's some detail as of 2012: https://storage.googleapis.com/pub-tools-public-publication-.... Note that host speed is 40Gbps. And here's FB talking about migrating from 40Gbps to 100Gbps in 2016: https://code.fb.com/data-center-engineering/introducing-back...