News for hackers. Stuff that matters.
Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by polskibus on Friday December 09, 2016 @05:28AM (#13137888)
Multicast has been used TIBCO Rendezvous to deliver high performance, high volume messaging in financial sector for ages. Can you elaborate on your claims?
Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by revertts on Friday December 09, 2016 @09:50AM (#13139019)
Messaging in the financial sector has very different requirements from messaging for a general service backend, which I assume is what the OP is referencing. As an example, a certain ecommerce site back in the day was heavily tibco based and migrated off due to inefficiencies at scale, multicast storms, debugability, and the general nature of multicast failures (for the most part you get no graceful degradation - which is fine when you have tight control of the network and do careful capacity planning, but isn't the mode that most tech companies live in).

Anecdotally, non-multicast setups are easier to scale in hardware (because you can push commodity farther) and most critically are easier for the normal type of engineer you'd find in the tech sector to reason about (because many aren't comfortable with networking equipment, so it's much easier moving that state out of the network and into layers they can easily debug).

Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by otterley on Friday December 09, 2016 @10:46AM (#13139342)
I worked on such a site in the mid-2000s. TIBCO Rendezvous was extremely efficient in the steady state, but boy, it had a catastrophic failure mode: during network congestion or receiver saturation, it would start repeating messages, causing the network congestion and saturation to worsen. The only way to stop the bleeding was to shut everything down.

At any rate, most cloud SDNs don't support IP multicast (unless you use an overlay like Weave), so it's sort of a nonstarter for them.

These days I might use Redis pub/sub along with a multi-level cache (e.g. LMDB + local Redis + remote Redis) if I need to distribute data to a large number of clients efficiently.

Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by tptacek on Friday December 09, 2016 @12:18PM (#13140027)
This is my experience with Rendezvous as well. The very largest financial organizations using Tibco tend to have a whole group of engineers whose only job is to keep Rendesvous from throwing a rod. Their networks are also overprovisioned and their traffic rates are pretty straightforward to characterize.

You can, of course, use Rendezvous without multicast.

And: does anyone still use it? The last exchange I worked with that did was migrating from it to JMS (which sort of gives lie to the idea that Rendezvous's multicast performance was that big a deal).

Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by otterley on Friday December 09, 2016 @01:58PM (#13140897)
Back in the 1990s and early 2000s, network bandwidth and server capacity was a lot lower, so IP multicast was the most efficient way to distribute data to hundreds or thousands of clients.

Now that 10Gb+ is commonplace and modern servers can service hundreds or thousands of message bus consumers with efficient kernel-managed event loops, yes, it's arguably less needed now. But nobody will claim it's efficient from a network perspective. :)

Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by tptacek on Friday December 09, 2016 @02:15PM (#13141016)
Depends on what you mean by "from a network perspective". Bandwidth? Sure. Network processing? Not so much: multicast is expensive for routers.
Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by SEJeff on Friday December 09, 2016 @03:05PM (#13141395)
Most financial firms (HFTs like I work for at least) purged Tibco with scorn and hellfire long ago due to reasons like the ones you just mentioned.
Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by polskibus on Friday December 09, 2016 @03:59PM (#13141799)
What did they replace it with?
Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by SEJeff on Friday December 09, 2016 @04:47PM (#13142134)
Non-open source proprietary systems, as one does :/
Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by kasey_junk on Friday December 09, 2016 @06:32PM (#13142829)
https://github.com/real-logic/Aeron

Is well regarded & open source

Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by SEJeff on Saturday December 10, 2016 @06:50PM (#13148681)
Several firms (not the one I currently work for) have implemented their own variants of NASDAQ's "MOLD" reliable udp protocol:

https://www.nasdaqtrader.com/content/technicalsupport/specif...

It is actually quite good.

Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by mrchicity on Friday December 09, 2016 @05:58PM (#13142621)
By general service do you mean a system where response time isn't especially important? Using TCP to fan-out messages in a PubSub pattern is going to waste network and compute resources in a serious way. It can easily exhaust capacity or result in convoluted, unreliable systems with layers of proxies for managing load.

FWIW when you talk about different requirements, I'm not sure you're correct. For traders, sure, oftentimes they can just dump old data if they fall behind because it's mostly irrelevant. However, stock exchanges have performance and reliability/availability as top requirements. They must record all transactions and can't crash or lock-up. Most use multicast internally and externally. They would never be able to scale and maintain a low response time with TCP.

Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by sseveran on Friday December 09, 2016 @10:33AM (#13139268)
If its the same ecommerce site I am thinking of, then there was also non-trivial message loss.
Re:Say No to Paxos Overhead: Replacing Consensus with Networ... by mritun on Friday December 09, 2016 @03:50PM (#13141721)
Yes, and with significant outages too. Again, I'm not saying multicast doesn't work. Multicast works well as long as you can guarantee that your network topology, components (switches and routers), servers and usage pattern will be predictable (and you test everything rigorously before going live). I can imagine that being true at some places with extremely tight focussed requirements.

In a dynamic environment where you'd be scaling continuously and adding components and servers and building new services, this becomes a non-starter. Multicast implementations doesn't fail gracefully -- there's very little warning before things come to a complete halt and need complete shutdown and reboot of entire network.