As a datacenter technician for Google, this makes me happy to see other people recognize this.

Could you shed some light on why google moved away from their shipping container design to a more traditional yet still custom data center approach?

Thanks!

Not working for Google, but I think it's for multiple reasons.

- hardware has gotten a lot faster, which helps the second reason.

- Google giving away free GGC nodes to ISPs (if you meet their requirements). The distribution of these nodes has gotten a lot wider since container days.

- Google is building more smaller scale (a dozen racks or so) traditional data centers closer to ISPs.

I don't think shipping container design was never a very good idea, to begin with - it was an interesting experiment that didn't scale as well as they thought it would.

It’s just sad that we’ll never see any of that openly available, or described.

It's not a lot, but there are a bunch of pictures on google.com/about/datacenters, including a 'street view' inside our DC in North Carolina.

There are also a bunch of papers on the network architecture, https://research.google.com/pubs/Networking.html .

But yes, a lot is closed for the obvious reason that this is really core to the business Google is running.

(I'm a datacenter engineer at Google)

There's a lot of openly available networking hardware on the Open Compute Project.

Sure, but that's not even close to the "every board and switch is custom designed" story.

there isn't as much exotic "stuff" as you might think. they have the same ODMs build switches and line cards to their specifications, with chips from the same vendors (Broadcom, Cavium, etc) only they can easily hire a team and sign the NDA and write/optimize/perfect the forwarding plane and control plane implementations.

hell, the IS-IS implementation in Quagga is usable because of Google :)

Disclaimer: I'm an SRE at Google, my opinions are my own.

We actually release a decent amount of information, see [0]. I particularly recommend "Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network" [1].

[0] https://research.google.com/pubs/Networking.html [1] https://research.google.com/pubs/archive/46286.pdf

Absolutely. The servers in their production racks are nearly 100% custom down to the silicon and involve serious amounts of engineering to make happen.

Just curious - How are their servers different than FB's who also modified the heck out of them?

Sorry but I have to disagree. They are custom but that kind of customization is not all that difficult. Essentially it's using same HW as everyone else in a way that supports their scale better. That's it.

Now I really don't know anything about this, but I remember quite a while ago when configuring a Linux kernel build that there was an entire section dedicated to Google's server hardware. If it's really all that normal, why would it be Google-specific/why wouldn't there be other large vendors present as well? Talking about the GOOGLE_FIRMWARE menu.

But not custom enough for Intel to remove Management Engine from their CPUs

> Pixel is a funny hobbby for Google

Seems like it but unfortunately it is their product and they are selling it.