How do you load balance? i mean load balance the "public ip"

In some networks DNS failover is really not that great, so at least a virtual ip needs to be used.

We use haproxy. I wrote code [1] that configures it based on Consul to do load balance. It has been running in production for 2 years without issue (tested with consul 1.0.6)

For people wondering why not use consul template, this has benefit of understanding haproxy, and it minimizes numbers of restarts to make changes.

Using haproxy over and has the benefit that if Winkle or consul goes down things continue to work, just not updates.

[1] https://github.com/takeda/winkle

Haproxy only solves a single Part of the Problem. If you do DNS based failover you should really Check how clients behave, when one node goes down. Without a floating ip or a cloud lb, some stuff will be troublesome

The haproxy method doesn't rely on DNS at all so I'm a bit confused.

well either it uses DNS for failover or you have ipvs (lvs, keepalived) enabled or worse if the machine with haproxy crashes your basically dead. Of course there is also bgp and anycast, but this is not "cheap"

That's not how it works. It is very similar to sidecar approach which various service discovery solutions have. You have haproxy running locally on localhost and you communicate with it, the haproxy then routes the request to right nodes. No DNS, no LVS and no keepalived.

well I've talked about edge Load Balancing..

Scalable NFS, riiite.

If you have some time to read "how Google works" you would be surprised by how long the company ran on NFS. I assume there are lots of workloads running on Borg to this day on top of NFS. If that isn't enough for you you should have a look in the client list of Isilon and see which kind of work they do, in case you ever attend a SIGGRAPH most of what you see is built on top of NFS, so, essentially, all of the computer graphics you see in movies. At last job our NFS cluster did 300 000 IOPS with 82gb/s throughput

82gb/s (assuming you mean gigabit) is _per-node_ throughput at Google (or FB, or I assume Amazon/Microsoft -- they all use 100GbE networks now). 300K IOPS is probably per-node, too, at this point. :-)

Having a 100gbps nic in a node isn’t the same thing as doing storage at that speed in an HA cluster.

Also, don’t confuse 100 gbe networks where spine links are 100 but the node links are only bonded 10s (much more common at $fang).

Nope. It's all 100GbE throughout as far as I know. And people do work really hard to be able to saturate that bandwidth as it is by no means a trivial task to saturate it through the usual, naive means without the use of RDMA and Verbs. Years ago when I was there it was (IIRC) 40Gbps to each node straight up.

It's a necessity really. All storage at Google has been remote and distributed for at least the past decade. That puts serious demands on network throughput if you want your CPUs to actually do work and not just sit there and wait for data.

Here's some detail as of 2012: https://storage.googleapis.com/pub-tools-public-publication-.... Note that host speed is 40Gbps. And here's FB talking about migrating from 40Gbps to 100Gbps in 2016: https://code.fb.com/data-center-engineering/introducing-back...

Sorry I don’t have to read it because i was borg sre for 6 years and i know how (the server part of) it works. You assume wrong.

I know there are a lot of companies that try to put some lipstick on nfs pig and call it reliable/scalable/etc. so long their clients don’t actually try to run it at scale or don’t complain too publicly when they try and can’t, they are able to get away with it.

Your concept of what is scale looks very different than mine, in my experience NFS does a very good job for in-datacenter workloads. CG rendering, oil/gas and others usually take this approach for HPC as far as I've seen. I consider this "scale". Close to 100k procs sharing the nfs is the biggest cluster I've worked at.

Of course that over longer networks it isn't suitable as the roundtrips have too much latency, other than that, is your experience much different regarding nfs?

What you consider ‘scale’ is a high watermark used by cloud providers that is irrelevant to 99.999% of the industry.

Supporting all of a Fortune 500’s business operations is very reasonable to call ‘scale’ in the normal world.

Your comment is a like a billionaire claiming that somebody that managed to hit 30 million isn’t rich.

worked at a company with 4 Petabytes on NFS ... FWIW