News for hackers. Stuff that matters.
Re:System design implications of declining network latency v... by chollida1 on Monday November 02, 2015 @08:17AM (#10491641)
> Since 1995, we’ve seen an increase in datacenter NIC speeds go from 10Mb to 40Gb, with 50Gb and 100Gb just around the corner.

This has been a huge factor for our firm.

Trading systems used to be, by necessity, monolithic in design. Your risk system, feed handlers, portfolio management, strategies, execution management, and drop copy/logging systems were all built into one process due to poor speed of interproceess communication, even on the same vlan. You just couldn't get the required performance and certainty with a distributed system. I mean its nice that your distributed database is eventually consistant over the course of 500 milliseconds, but that's a couple of order of magnitude too slow

Now, networks are sooooooo fast and bandwidth is so plentiful that we can afford to split many of the major subsystems onto different machines.

It's important to note that we don't split these subsystems up for redundancy, I mean we aren't Netflix, we can't pull out major subsystems and keep trading. if one goes down we intentionally take down the entire system. You can laugh at me here if you'd like....

Rather we split the systems up so that we can test and upgrade them easier. Before the split, each upgrade was a major change that might have happened 2x a month, now we can upgrade individual systems 2-3 times a week. Agian, you may laugh now, but I put a very big premium on stability and certainty.

I remember talking to someone at Google who told me that their low latency RPC framework was considered to be one of their key technological advantages. Having made the move over the course of a couple of years, I'd be inclined to agree, a robust and fast RPC framework is an incredibly freeing piece of technology.

Re:System design implications of declining network latency v... by MichaelGG on Monday November 02, 2015 @01:52PM (#10494080)
If a single scaled up server can actually handle the load, why introduce a network? Obviously cross-network comms can't be faster than cross-process comms. And with current NUMA machines, you have a system that's really more like a cluster of computers anyways (with a fast inter-proc but still slower than intracore). What's the real benefit of using RDMA or something to write to a foreign machine's memory versus nontemporal writes to a foreign socket's memory?

BTW any suggested reading on the technical implementation design of trading systems? I read that Island's system was distributed (horizontally scalable) even in the 90s. Is that doing things like running separate trading instances per symbol? (Which should work unless there's atomicity required across symbols, like in spread orders?)

Re:System design implications of declining network latency v... by openasocket on Monday November 02, 2015 @05:11PM (#10495710)
I have no idea about the OP's system, but I can see some advantages to breaking components up across a network, at least for vertical scaling. First is that server prices do not scale linearly: four 10-core machines will generally be cheaper than a single 40-core machine. Second is that some programs can hog memory/CPU time; forcing programs to run on their own machine can ease some of that contention. These gains can seem rather small, but can make a big difference at large scales. If you don't mind the drop in latency it's a perfectly acceptable architecture.
Re:System design implications of declining network latency v... by noir_lord on Monday November 02, 2015 @07:05PM (#10496394)
Third advantage is it forces proper separation, it can be very tempting to do things you shouldn't to local resources for "speed".
Re:System design implications of declining network latency v... by Florin_Andrei on Monday November 02, 2015 @01:27PM (#10493880)
> You can laugh at me here if you'd like

Why laugh? Different businesses, different requirements, different solutions. Horses for courses, as they say.

Re:System design implications of declining network latency v... by footiethrowa on Monday November 02, 2015 @09:54AM (#10492049)
> I remember talking to someone at Google who told me that their low latency RPC framework was considered to be one of their key technological advantages. Having made the move over the course of a couple of years, I'd be inclined to agree, a robust and fast RPC framework is an incredibly freeing piece of technology.

Have you found Google's RPC framework to be robust and fast, or something else? And if so, would you mind sharing what it is? Thanks!

Re:System design implications of declining network latency v... by chollida1 on Monday November 02, 2015 @12:11PM (#10493179)
We've experimented with Protocol Buffers, Thrift and CapnProto.

We settled on protobuf but capn proto was super fast. I just didn't have the time to play around with it or fortitude to put it into production with it being so new.

I should point out, that's my own hang up and not mean to disparage Capn Proto at all, call it the web 2.0 of "Nobody ever got fired for buying IBM".

Re:System design implications of declining network latency v... by kanslanf on Monday November 02, 2015 @12:27PM (#10493350)
I really like what I've read of CapnProto and would try it for my next distributed system project.
Re:System design implications of declining network latency v... by corysama on Monday November 02, 2015 @03:48PM (#10495101)
CapnProto has good competition from Google's Flatbuffers. Which one is better comes down to the specifics of your data.

https://google.github.io/flatbuffers/

https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-...

https://news.ycombinator.com/item?id=7901991

Re:System design implications of declining network latency v... by rictic on Monday November 02, 2015 @12:15PM (#10493220)
The newest version was actually open sourced earlier this year: http://www.grpc.io/
Re:System design implications of declining network latency v... by _yy on Monday November 02, 2015 @12:05PM (#10493120)
Protobuf, probably.