News for hackers. Stuff that matters.
Re:MySQL High Availability at GitHub by merb on Wednesday June 20, 2018 @03:57PM (#17359151)
I can understand that VIP's can sometimes be akward, ARP fails or even worse that VIPs won't be correctly released due to unknown stuff.

However I do wonder why they choose such a complicated thing instead of looking into something like BGP that would've replaced their ARP setup.

Also they should've probably tried to look into K8s since inside a cluster such problems do not exist. (There you have the problem to have a good LB at the outside of your cluster)

Re:MySQL High Availability at GitHub by theptip on Wednesday June 20, 2018 @04:27PM (#17359442)
> something like BGP that would've replaced their ARP setup

Just a guess, but anecdotally it seems to me that inside a modern mid-size dev org there are a lot more engineers familiar with K/V stores like etcd/consul/ZK than IP routing protocols like OSPF and BGP.

It's not necessarily the wrong decision to forgo an existing well-tested tool in favour of a hand-rolled solution, if you don't have any experts on that existing tool in your org.

Re:MySQL High Availability at GitHub by mschuster91 on Wednesday June 20, 2018 @05:16PM (#17359855)
> It's not necessarily the wrong decision to forgo an existing well-tested tool in favour of a hand-rolled solution, if you don't have any experts on that existing tool in your org.

It will, however, lead to other problems, especially:

- reinvention of the wheel (aka: time waste)

- starting from scratch: technologies like OSPF, BGP, or (when we're at higher levels) my/pgSQL vs noSQL are sometimes decades old and battle tested. Which means that there are a lot more edge cases that have been discovered and fixed/documented, as well as best practices that have evolved. Same goes for security. Or even fundamental stuff like ACID criteria (which many of the noSQL friends have had to re-discover and learn)...

There's a reason yuge enterprises and government offices still run on mainframes (or, mainframe software on modern hardware): the stability and resilience of these beasts with uptimes in year-scales is something the hipster crap crowd is enviously looking at. And there's a reason that redesign projects in these areas fail so spectacularly: there are decades worth of domain knowledge and experience encoded in these systems. It's impossible to recreate this in a year or two, especially when outsourcing the work to body shops.

Only thing I wonder is: it should be, by now, common knowledge (or: state of the art) that outsourcing yuge projects to body shops, especially foreign ones, leads to disasters. So why are managers and politicians not prosecuted or otherwise held responsible when screw-ups originate? After all, when one e.g. designs a building that violates fire code and people die as a result, or one grossly ignores security best practices, the responsible persons can get charged, too...

Re:MySQL High Availability at GitHub by zzzcpan on Wednesday June 20, 2018 @07:31PM (#17360840)
> my/pgSQL vs noSQL are sometimes decades old and battle tested

> run on mainframes (or, mainframe software on modern hardware): the stability and resilience of these beasts with uptimes in year-scales is something the hipster crap crowd is enviously looking at

Why so much ignorance? Distributed systems are not hipster things, it's an actual science that helps to understand why no amount of battle testing can make this old tech resilient, it's fundamentally broken. There is just no resilience without at least proper distributed algorithms.

Re:MySQL High Availability at GitHub by justinsaccount on Wednesday June 20, 2018 @05:21PM (#17359900)
Re:MySQL High Availability at GitHub by scurvy on Thursday June 21, 2018 @12:35AM (#17362193)
+1 for BGP/OSPF-based VIPs in a layer 3 network. It's 2018. No reason why you can't easily run a layer 3 network with rich routing protocols. Super easy.

You don't need application level "discovery" when the IP never changes; the network will tell you where it currently lives.

I'll take a layer 3 network over raft/consul any day and all night.

Re:MySQL High Availability at GitHub by voltagex_ on Thursday June 21, 2018 @01:21AM (#17362338)
I don't think I've ever heard BGP described as "easy", but then again it's not really my area.

Where could I go to read up on this without being a networking expert? Are there any safe ways to play with BGP in a lab environment? How expensive would said lab be to set up?

Re:MySQL High Availability at GitHub by merb on Thursday June 21, 2018 @04:36AM (#17363190)
Are there any safe ways to play with BGP in a lab environment? actually there are multiple ways of running bgp in a lab. there are cheap variants that are routed into the internet with ipv6 and there are ways to just play without without talking to the outside network.

How expensive would said lab be to set up? it heavily depends on what you want. for a high available lab you would of course need multiple nodes/routers, depends on what you want to do.

for just playing with bgp, actually using bird http://bird.network.cz/ is good enough.

Actually ripe sometimes publishes articles with BGP. Here is one that sets up a ANYCAST routing with BGP https://labs.ripe.net/Members/samir_jafferali/build-your-own...

Re:MySQL High Availability at GitHub by scurvy on Thursday June 21, 2018 @08:11PM (#17370236)
Juniper and Cisco have lab versions of their products (Olive, vMX, vSRX) for free. Arista has something, too.

BIRD is super easy to setup, too.

Re:MySQL High Availability at GitHub by voltagex_ on Thursday June 21, 2018 @08:30PM (#17370322)
If I'm not studying for any particular cert (I'm just looking to fill some gaps in my knowledge) then it looks like BIRD is the way to go.
Re:MySQL High Availability at GitHub by scurvy on Friday June 22, 2018 @12:20AM (#17371270)
bird is the word
Re:MySQL High Availability at GitHub by scurvy on Thursday June 21, 2018 @08:16PM (#17370256)
Network people who understand BGP are cheaper and easier to find than someone who actually knows how Consul/raft works. You might think you know how consul works, but do you really? Consul and raft have caused several high profile outages -- including at places full of smart people like Amazon. If Amazon can't do it right, what chance do you have?

Especially when there are several-year old bugs open against consul, where it just completely loses consensus and can't deal with losing a single node(0)

0: https://github.com/hashicorp/consul/issues/993#issuecomment-...

Re:MySQL High Availability at GitHub by voltagex_ on Thursday June 21, 2018 @08:29PM (#17370318)
This is good to know if I ever find myself working at this scale.

I don't have skin in the game but it makes sense that a much older, tested technology would have more people who know it well.

Re:MySQL High Availability at GitHub by ton31337 on Thursday June 21, 2018 @10:30AM (#17365464)
that exactly what we are doing at Hostinger ;-) this one http://blog.donatas.net/blog/2017/02/26/exazk/