I was brought up to only use stonith via serial or other non-switched network connection. Running over the same network is bound to cause problems. But its not a great solution anyway.

We used to STONITH (with an APC network power switch) over serial. It didn't really help because serial ports have really bad quality control, and lose a lot of 'packets'. :)

We ended up with dedicated serial cards, and multiple network links just so STONITH worked properly. And even then, Heartbeat was buggy as hell back then, so we'd end up in a active-active situation way too many times and we'd end up with FS corruption.

We were running Reiser on DRBD (ahh, the good old days). Had to hack the kernel a fair bit to make it all work.

Anyway, in the end, we just abandoned the automated Heartbeat failover stuff, and just used it to alert us. We manually ran the failover scripts when a human determined that there was in fact a real failure.

It's nice that modern servers often come with 4 NICs, and Gigabit is Auto-MDIX.

Agreed. I completely agree, but the world isn't as kind. Providers often have funny rules about how you can cable things up in their datacenters, and as noted in #4 of where GitHub goes from here that needs to be addressed.

What do you suggest as a replacement? Heartbeat is in use at many big players.

I don't know how good Heartbeat is these days. I'm just going based on our experience back then and GitHub's stated experience now.

We (in a new place) use our own fail over code now, but we're running databases instead of file servers -- and you usually don't want to STONITH a database.

For file servers we use Gluster, and it works great for us (I'm sure there are reasons GitHub are doing active-passive DRBD) and you don't need STONITH since it's shared-nothing.

There's still plenty of things to keep in mind with Gluster, though. If you use a replicated setup for redundancy, and you get a split network, you can very well end up with an inconsistent state because you can get into situations where you write to different replicas from different clients and replication doesn't succeed.

Gluster will just throw it's hands up during self-heal if that happens, and you'll need to manually resolve it.

Your filesystem structure won't break, but your files certainly can.

I'll just point out that experience with heartbeat from 2001 is rather outdated.

That would be similar to comparing 2.4 kernels problems to the most recent 3.7 kernel. Likely not an overly useful anecdote.

For the record we use heartbeat at work with no issues such as this.

While I agree with you, I think the anecdote is still useful because it shows that the problems of the present are really problems of the past too.

Software quality improves, sure, but you can still learn from the past.

I'd say its more an implementation issue than software. And that this is a solved issue even 10+ years ago with heartbeat.

Our heartbeat links are segregated from the public network with separate network cards/switches. So the specific issue github hit here isn't what we would have encountered. We do have quad nic cards for a reason in the systems we run, this issue github hit is one of the various reasons you don't run heartbeat over the public topology. It will bite you in the ass no matter what cluster software you are running.

Unless you also have a disk heartbeat over shared fibre/scsi, or maybe serial but same difference. Depending upon the public network though is a lost cause.