News for hackers. Stuff that matters.
Re:Kubernetes on Google, Azure and AWS Compared by stevenacreman on Tuesday October 09, 2018 @03:59PM (#18179345)
Hi, there's a comment on the cross region networking for GKE.

"Each cluster receiving an IP range for nodes and another for the containers inside, which are directly routable across your private network, other clusters, and regions."

AWS and Azure don't have a flat global network. You have to setup VPN's and complicated overlay networks.

Re:Kubernetes on Google, Azure and AWS Compared by halbritt on Tuesday October 09, 2018 @11:11PM (#18181677)
Couple things:

Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well.

Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance.

On workloads where I care about Network IO:CPU ratio, I'm using 8 core nodes and seeing 16gbps throughput between them consistently.

Re:Kubernetes on Google, Azure and AWS Compared by gazoakley on Wednesday October 10, 2018 @04:14PM (#18188450)
Re:Kubernetes on Google, Azure and AWS Compared by ti_ranger on Wednesday October 10, 2018 @01:51AM (#18182331)
> Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well.

AWS tries to avoid letting customers create multi-region or global failure modes, like:

* https://status.cloud.google.com/incident/compute/18005 22-hour incident with GLOBAL impact resulting in many VMs getting duplicate IPs (this no network connectivity) on 2018-06-15

* https://status.cloud.google.com/incident/cloud-networking/18... 3-hour (seemingly GLOBAL) impact to load-balancers (no updates or creations) on 2018-01-03

* https://status.cloud.google.com/incident/compute/16007 - 18-minute GLOBAL outage on 2016-04-11

* https://status.cloud.google.com/incident/compute/15055 - 5-minute GLOBAL outage on 2015-08-04

* https://groups.google.com/forum/#!topic/gce-operations/fynnX... 43-minute multi-region packet loss due to multi-region configuration deployment on 2015-03-07

* https://groups.google.com/forum/#!topic/gce-operations/1uw-q... unscheduled reboot of 28% of instances in multiple regions in ~1.5h on 2014-09-17

There have been recent changes in many AWS services (e.g. DynamoDB, S3, Aurora) to allow cross-region replication without introducing any multi-region dependency, precisely to make it easier to implement multi-region infrastrucure to tolerate a single-region failure (however rare that is on AWS compared to global outages on GCP).

> Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance.

Which instance types did use on AWS? Many recent instance types (r4, i3, c5, r5, z1) support up to 10Gbps with once core (2 vCPU instances). However, you may need to use placement groups ( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placemen... ) in large regions in order to get the full throughput on low TCP connection numbers. The only reason I can think of that would explain why GCP doesn't have this problem is they don't have any regions anywhere near as large as large AWS regions ...

Edit: white-space only formatting changes

Re:Kubernetes on Google, Azure and AWS Compared by halbritt on Thursday October 11, 2018 @01:52AM (#18191212)
Your point about global failure modes is well-taken. Even still, one could reasonably expect inter-region networking not to be so difficult.

As for the instances I tested, they were m4 and r4. To get full 10GE out of either, I needed to use m4.10xl and r4.8xl, both roughly around $900 per month.

By comparison an n1-standard-8 is about $60/mo and the n1-highcpu-8 is $50/mo. I've tested the former and got something like 15.8 or 15.9gbps using iperf.

IMHO, that's pretty significant.

Re:Kubernetes on Google, Azure and AWS Compared by shshhdhs on Tuesday October 09, 2018 @05:08PM (#18179823)
> AWS and Azure don't have a flat global network. You have to setup VPN's and complicated overlay networks.

AWS has cross-region VPC peering, which is simple to setup and makes it feel “flat”. There’s no need for VPNs or overlay networks.

Re:Kubernetes on Google, Azure and AWS Compared by nodesocket on Tuesday October 09, 2018 @05:52PM (#18180132)
Until you have to reference security groups across the peering connection. You can't reference a different region's security group id as a source, thus your forced to just whitelist the entire subnet/cidr.

Also, while setting up peering is a 10-20 minute task there are lot of constraints, such as the VPC's have to have different CIDR ranges. Oops, if you go with the AWS default, probably all regions using 172.31.0.0/20.

Google got networking right with a flat global any region setup. If you need segmentation then just create a new GCP project. If it turns out projects need to communicate after all, GCP has VPC network peering.

Re:Kubernetes on Google, Azure and AWS Compared by p0rkbelly on Wednesday October 10, 2018 @12:52AM (#18182050)
You can use PrivateLink for this instead of peering. That way you just hit a local IP address/ENI in your VPC. CIDR Ranges don't matter with that. (not sure about the x-region story)

Two different design philosophies, there are been a global GCP networking outages due to having the single flat network. A few this year. All providers have had their issues, but, global blast radiuses concern me.