That isn't really how production networks work in my uneducated opinion. If they are connected to the production network then they are the production network, and the level of isolation required to make that not the case would be so extreme as to make things potentially more unreliable.

Others can correct me if I'm wrong about this. All I know is that the production network where I work is not air gapped in the way that would be required to truthfully consider testing networks a non production environment, so non prod changes typically wind up in front of the change review board anyway.

Ask your own sites network engineers and see if they have similar constraints because I would be interested to hear more perspectives on that.

One other thing I will say is that the abstractions of "config plane" and "data plane" and "control plane" don't really exist on real physical systems. That is mostly an abstraction created for applications people, those systems are not going to be totally blocked from interacting with eachother, they kind of have to. So if any of your "planes" are shared with production it is a production environment.

That would mean that all networks which peer with the Internet would necessarily be considered Production. This isn't that reasonable outside certain niches (i.e., national government networks).

Instead, what's commonly done is to provide a Controlled Interface (to borrow a term from those national government networks) that gates which things are at which level of trust. This is where security boundaries are enforced -- and if they are sound security boundaries things on either side can't reasonably damage the other side.

That's super interesting, and you're definitely right about the internet thing. I suppose our network guys must have some way to see if a change will propagate beyond a particular interface?

> One other thing I will say is that the abstractions of "config plane" and "data plane" and "control plane" don't really exist on real physical systems

If you use any sort of virtualization: the control plane (infra) vs data plane (apps) will naturally evolve from the architecture. The config plane and control plane can get squashed into the same thing though, but it can also be disparate for at both infra- and application level.

Data plane and control plane are definitely a thing in real physical systems- look at a classical router, where the packet processor works independently of, and is occasionally programmed by, or assisted by, a message passing from the data plane to the control plane. That control plane is typical elsewhere on the main board, talking to the data plane through a well-specific protocol.

Google's network is complicated, making many assumptions about "what is prod" etc hard to reason about.

I've seen a couple companies (credibly) claim multiple independent networks, but it seems to be pretty expensive to do in practice. There's a lot of cost to be saved by having a single network, and it's too tempting to share things making the independence illusory.

Probably you get better independence with smaller providers where the provider isn't yet big enough to actually run a global network. Then, each PoP is likely to run or fail independently. Google and peers have enough scope that they can run a single global network where one can experience the Haiku:

It’s not BGP There’s no way it’s BGP It was BGP

(DNS also fits in there; any sort of automation to push DNS or BGP changes will probably fit, if the name was carefully chosen)

Google does have this, but one of the networks is way bigger than the other, so they can't exactly fail B4 over to B2.

This "one global network" instead of, say, AWS's regions, was a selling point for Google against AWS. Big enterprises loved loved LOVED not having to think in terms of regions.

Recently, AWS seemed to be finally capitulating to sales demand and globalizing things.

I wonder if that will slow now there's a competitor incident report with the answer being region sharding.

> Recently, AWS seemed to be finally capitulating to sales demand and globalizing things.

Short answer: no. cross-region services are few and far between.

Long answer: https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso...

While correct that few services out of the total are cross-region I think only a few need to be to get the bulk of functionality.

If restoration(S3 & snapshots) and networking are both cross region it is possible to avoid outages without much effort.

S3 already supports global endpoint, cross regional replication and even manual failover.

RDS supports cross regional snapshots.

Networking has a bunch of options for crossing regional now.

If you're worried about AWS global endpoints and services going down the design is always around pointing to the global but being able to fall back to a specific region up.

At this point can just run EKS in N regions, have the clusters stateless and treat RDS+S3 for state.

Note that the region sharding is just about the config push, the network itself isn't getting shareded. The goal is to ensure a bad config push can't impact >1 region (in theory), not to segregate the actual network.