News for hackers. Stuff that matters.
Re:LSE Halts by ig1 on Friday February 25, 2011 @06:31AM (#2261840)
The problem is that the LSE can't hire top tier talent. There are plenty of people who are experts in building hugely concurrent and fail-safe exchange software, and they pretty much all work for investment banks.

The LSE can't afford to compete with $1000/day salaries so they end up hiring second-tier developers. And unfortunately second-tier developers are a lot lot worse than first-tier developers.

Re:LSE Halts by laktek on Friday February 25, 2011 @06:53AM (#2261872)
I don't think it's about talent, but more of a problem of proper planning and execution. It seems things had been rushed to meet the tight demands, causing the chaos.

When LSE acquired MilleniumIT, to redo the platform, they claimed to have developed world fastest trading platform[1] and had enough talent under their belt developing trading systems around the globe for last decade.

[1] http://www.wallstreetandtech.com/exchanges/showArticle.jhtml...

Re:LSE Halts by ig1 on Friday February 25, 2011 @07:17AM (#2261912)
None of the exchanges have the required talent. Being the best exchange technology company is like being the best team in a minor league. If you're good at extreme low latency stuff you're going to be working for a bank.

At a bank being great at low-latency can mean tens of millions of dollars of profit for your trading desk and a huge bonus for your personally.

At an exchange being great a low-latency means ... well not much, your exchange gets a bit more flow and you get a pat on the back.

Re:LSE Halts by chollida1 on Friday February 25, 2011 @10:14AM (#2262403)
> At an exchange being great a low-latency means ... well not much, your exchange gets a bit more flow and you get a pat on the back.

Exchanges pay out bigger bonus' than you'd imagine. 7 figures for a programmer is high but no where out of the ordinary. Says a guy who has worked for an exchange.

Re:LSE Halts by ig1 on Friday February 25, 2011 @11:12AM (#2262597)
I seriously doubt any exchange is regularly paying developers multi-million dollar bonuses.

Which exchange did you work for, I don't see one on your linkedin profile?

Re:LSE Halts by _debug_ on Friday February 25, 2011 @11:49AM (#2262764)
Exchanges often use (relatively) esoteric hardware + real-time OSes, such as the Tandem NonStop series. Experience with such a platform + domain knowledge about the finance industry is a rare combination, so I find it plausible that a couple of core expert programmers on the stock exchange's system get compensated quite well. High six figures + 80% bonus == seven figure, I guess?
Re:LSE Halts by chollida1 on Friday February 25, 2011 @03:48PM (#2263730)
> I seriously doubt any exchange is regularly paying developers multi-million dollar bonuses.

I'd agree with that. Having said that, I doubt many investment banks are regularly paying developers multi-million dollar bonuses.

The people who get those are called quants or traders, even if they have programming skills, just my experience.

> Which exchange did you work for? I'd rather not say, if you want to know, contact me via email. Not sure how you got my linked in info as it's not attached to my profile.

Re:LSE Halts by rs on Friday February 25, 2011 @12:22PM (#2262911)
None of the exchanges have the required talent.

Ahem, based on what evidence exactly did you reach that conclusion ?

TBH, from all the exchanges/brokers I've seen and dealt with in the past, there are a few that are pretty solid - Deutsche Borse is one of them. There's EuroMTS as well.

Essentially, most exchanges that deal with many market makers have got an extremely challenging technical problem to solve - scaling and latency. One that most investment banks can't even begin to solve.

I'm not saying there aren't any good talent in banks - there are plenty. I'm just saying that there's some really smart, talented people in exchanges.

Re:LSE Halts by alecco on Friday February 25, 2011 @07:26AM (#2261929)
That's very likely. Any programmer with even an entry level position at the LSE would get head hunted by multiple agents in the first month. And the investment banks do pay £700/day for talent.
Re:LSE Halts by fendale on Friday February 25, 2011 @09:32AM (#2262265)
I know at least two guys who are getting paid closer to £1000 a day to work in London for a bank. The money is pretty much insane imho, and the guys tells me the 'talent' is just like anywhere else - ie most people are meerly compentent (but still getting paid £500 plus a day). Even graduates with zero experience are taken on at £300 a day or more.

He seems to better than competent, and so has managed to negotiate a much higher rate.

Re:LSE Halts by famoreira on Friday February 25, 2011 @04:31AM (#2261650)
Does anyone know if this halt is related to the recent switch over to the new Linux based platform? (http://news.ycombinator.com/item?id=686116)
Re:LSE Halts by known on Friday February 25, 2011 @06:46AM (#2261868)
I believe this is due to a glitch in the Application and not in Linux per se.
Re:LSE Halts by drstrangevibes on Friday February 25, 2011 @04:37AM (#2261658)
it is a new platform, hard to tell but I wouldnt go so far as to say linux is to blame, its more likely to be exchange software, they've gone from native to fix 5, so its using a new protocol, new os, new software stack and most likely new hardware stack
Re:LSE Halts by tallanvor on Friday February 25, 2011 @06:21AM (#2261827)
I don't think famoreira was trying to blame Linux. --That's just the part of the new system that you can reference easily.
Re:LSE Halts by famoreira on Saturday February 26, 2011 @02:45PM (#2266328)
Yes, I didn't mean to blame Linux. I was just referring to the system in the way it was made famous (replacing a Microsoft based system with a Linux one).
Re:LSE Halts by IDisposableHero on Friday February 25, 2011 @06:24AM (#2261829)
I don't think that "linux is to blame" or even that any choice of programming languages or other tools are to blame.

It's a large system with extra-ordinarily tight performance requirements, and a lot of integration with external systems, and by nature hard to phase in gradually. That kind of system is very hard to get right first time regardless of your toolset. I'd be interested to hear more about thier methods to deal with these requirements and risks, less about thier platform choice. IIRC, both this and the previous failed system were outsourced, which doesn't exactly fill me with confidence.

Re:LSE Halts by mtkd on Friday February 25, 2011 @06:24AM (#2261830)
"And as a final note, solely for the seriously nerdy techies, there's been some talk about the failure being due to F# error."

http://ftalphaville.ft.com/marketslive/

Re:LSE Halts by timrobinson on Friday February 25, 2011 @06:38AM (#2261846)
I've no idea where they got this from, although I mentioned it on Twitter as a joke: https://twitter.com/tim_g_robinson/status/41086872876892160
Re:LSE Halts by alecco on Friday February 25, 2011 @07:10AM (#2261899)
I still don't understand how they choose languages with garbage collection for systems that should be real time or similar.
Re:LSE Halts by davidw on Friday February 25, 2011 @07:37AM (#2261950)
Erlang is not "real time" in the hard, embedded systems sense, but it does pretty well and has GC.

Here's a bit more information: http://www.erlang.org/faq/academic.html#id55755

I don't know what sorts of 'real time' properties trading systems are required to have - can anyone talk about it with a bit more in-depth knowledge?

Re:LSE Halts by IDisposableHero on Friday February 25, 2011 @08:49AM (#2262139)
I don't know much, but for high-frequency trading, fast is never fast enough. response times are measured in microseconds.

http://richg74.wordpress.com/2010/10/25/speedy-new-lse-tradi...

http://www.h-online.com/open/news/item/Linux-boosts-Stock-Ex...

Re:LSE Halts by alecco on Friday February 25, 2011 @08:59AM (#2262163)
Speed isn't all. The system should be based in algorithms with manageable worst case time.
Re:LSE Halts by alecco on Friday February 25, 2011 @08:55AM (#2262153)
I don't buy Erlang camp's reports.

A GC will always trash the cache, and that's the worst issue right now in hardware. (If you take out HDD I/O.)

Re:LSE Halts by axod on Friday February 25, 2011 @07:24AM (#2261925)
GC isn't a problem for realtime systems, if configured correctly. For example, have the GC running incrementally in a separate thread.
Re:LSE Halts by CountHackulus on Friday February 25, 2011 @11:08AM (#2262584)
Close, you do need to stop the world every once in a while to GC some of the harder stuff.

But it's true, there's flavors of Java that are certified for hard realtime. If you stop the GC for 100ms every 2s regardless if you need it or not, that's pretty nice and predictable behavior for a realtime system.

Re:LSE Halts by wingo on Friday February 25, 2011 @11:18AM (#2262636)
100ms is an eternity. You can do GC with 1ms pauses.

http://www.infoq.com/articles/azul_gc_in_detail, for one example.

Re:LSE Halts by timrobinson on Friday February 25, 2011 @07:11AM (#2261900)
It's not F# or Java, it's C++. http://news.ycombinator.com/item?id=2261896
Re:LSE Halts by alecco on Friday February 25, 2011 @07:20AM (#2261919)
It could be some left over system from the previous .Net platform causing a knock-on effect.
Re:LSE Halts by IDisposableHero on Friday February 25, 2011 @07:29AM (#2261935)
It could be anything. Why do you want it to be that?
Re:LSE Halts by alecco on Friday February 25, 2011 @07:34AM (#2261945)
I don't want it to be that. I'm saying the rumour sounds plausible.
Re:LSE Halts by IDisposableHero on Friday February 25, 2011 @08:50AM (#2262141)
Not so plausible to me, considering that the old system was in C#.
Re:LSE Halts by dagw on Friday February 25, 2011 @09:39AM (#2262282)
I'm pretty sure I've read somewhere that they where using F# for some parts.
Re:LSE Halts by IDisposableHero on Monday February 28, 2011 @10:25AM (#2271489)
Can we have a link please?
Re:LSE Halts by tzs on Friday February 25, 2011 @12:46PM (#2263017)
That seems unlikely. F# only recently moved from being an experimental language from Microsoft Research to a supported .NET language. The previous system is old enough to predate that.
Re:LSE Halts by StrawberryFrog on Friday February 25, 2011 @05:31PM (#2264054)
Agreed. and Anderson consulting using an experimental, non-obvious language? I don't see it.
Re:LSE Halts by scott_s on Friday February 25, 2011 @11:26AM (#2262660)
You may be interested in the Metronome project, which aims to provide a GC suitable for realtime use: http://domino.research.ibm.com/comm/research_projects.nsf/pa...

An academic paper about the project, Design and Implemention of a Comprehensive Real-Time Java Virtual Machine: http://domino.research.ibm.com/comm/research_people.nsf/page...

Re:LSE Halts by chollida1 on Friday February 25, 2011 @08:26AM (#2262076)
that link doesn't contain the text 'F#' any where on the page.
Re:LSE Halts by timrobinson on Friday February 25, 2011 @10:49AM (#2262510)
Re:LSE Halts by chollida1 on Friday February 25, 2011 @01:26PM (#2263195)
Thanks:)
Re:LSE Halts by drstrangevibes on Friday February 25, 2011 @06:39AM (#2261851)
F# on linux?
Re:LSE Halts by timrobinson on Friday February 25, 2011 @06:40AM (#2261854)
Re:LSE Halts by mtkd on Friday February 25, 2011 @06:40AM (#2261858)
I doubt they'd be using Mono - if it really is an F# issue it maybe indicates it's a failure in some of the legacy system they are replacing.
Re:LSE Halts by IDisposableHero on Friday February 25, 2011 @07:35AM (#2261946)
As far as I know, the previous system didn't use F#.

"written in C#" http://www.computerworlduk.com/news/open-source/3260727/lond...

Re:LSE Halts by viraptor on Friday February 25, 2011 @06:40AM (#2261856)
Via mono. If that's true... that would be an interesting news for the project (not because of the bug itself of course...).
Re:LSE Halts by timrobinson on Friday February 25, 2011 @06:44AM (#2261864)
If the LSE were using F# we'd have heard a lot more about it already. For instance, from Microsoft.
Re:LSE Halts by viraptor on Friday February 25, 2011 @06:50AM (#2261869)
Maybe not if they were using F# on mono. Their PR would have to work very hard to put this in a way that doesn't say "LSE is using our language on a foreign platform using a completely alternative implementation, which has nothing to do with us apart some ECMA specs - and they're still successful without MS involvement. Yes, we just confirmed that you can trust your highly critical .NET environment to the alternative versions of runtime."
Re:LSE Halts by IDisposableHero on Friday February 25, 2011 @07:08AM (#2261896)
They have publicly stated that the system is written in C++ on Suse Linux.

http://www.computerworlduk.com/news/open-source/3260727/lond...

Re:LSE Halts by alecco on Friday February 25, 2011 @07:21AM (#2261921)
I wouldn't believe 100% in those statements.
Re:LSE Halts by IDisposableHero on Friday February 25, 2011 @07:32AM (#2261942)
True enough. If the statement turns out to be not entirely correct, that would be news.

I suspect that if there is any F# code, it's in a 3rd party system that talks to the stock exchange. There are lots of those due to the nature of an exchange, though they should all talk over the standard FIX protocol.

Re:LSE Halts by chollida1 on Friday February 25, 2011 @10:18AM (#2262415)
> I wouldn't believe 100% in those statements.

Why not? What makes you think this is a lie?

Re:LSE Halts by alecco on Friday February 25, 2011 @02:11PM (#2263380)
They systematically cover the problems and have zero transparency. Never a post-mortem. Like back in January.
Re:LSE Halts by timrobinson on Friday February 25, 2011 @07:00AM (#2261882)
OK, I wouldn't expect an official press release like this from Microsoft head office.

But the team in Microsoft Research are (a) keen to promote their language (people like Don and Brian often help early adopters personally) and (b) supportive of Mono (they've always released a .zip file for Mono at the same time as their Windows installers).

Re:LSE Halts by drstrangevibes on Friday February 25, 2011 @07:01AM (#2261883)
using a immature language (f#) over unproved software stack for an international multi billion dollar low latency transactions when there are dozens of proven highly stable functional alternatives that run well on a linux platform? I can hardly believe it
Re:LSE Halts by IDisposableHero on Friday February 25, 2011 @07:11AM (#2261901)
I don't believe it at all. Mainly because of the public statements that the new LSE trading system was written in C++.

On the other hand, it has to integrate with lots of enternal trading systems, one of which could easily be written in F#.

Re:LSE Halts by siika2000 on Friday February 25, 2011 @07:07AM (#2261892)
That's modern enterprise computing for you.
Re:LSE Halts by timrobinson on Friday February 25, 2011 @07:16AM (#2261909)
"Continuous trading has now resumed in all order driven trading services."
Re:LSE Halts by alecco on Friday February 25, 2011 @07:13AM (#2261904)
Sri Lanka? http://uk.reuters.com/article/2009/09/16/uk-lse-it-sb-idUKTR... (via /B @ftalphaville comments)
Re:LSE Halts by ashishb4u on Friday February 25, 2011 @04:47AM (#2261678)
Re:LSE Halts by alecco on Friday February 25, 2011 @06:28AM (#2261835)
What if all these systems were open sourced? Including all the load tests.
Re:LSE Halts by viraptor on Friday February 25, 2011 @06:54AM (#2261873)
At this size, some obfuscation might be good. If you find a bug and crash some browser, it's not a big deal. If you find some problem in LSE that can be triggered from outside, you're dealing with realworld $M transactions.

Also, unless you have a huge farm of servers it would be useless to you - and basically anyone really.

Re:LSE Halts by rbanffy on Friday February 25, 2011 @07:28AM (#2261932)
> What if all these systems were open sourced?

LSE considers its software to be a competitive advantage.

Re:LSE Halts by drstrangevibes on Friday February 25, 2011 @07:46AM (#2261968)
.....er sure
Re:LSE Halts by rbanffy on Friday February 25, 2011 @08:49AM (#2262136)
When they make it work, it could be ;-)
Re:LSE Halts by tpatke on Friday February 25, 2011 @06:08AM (#2261809)
Where is the fail over? If ever a system needed 99.9999...
Re:LSE Halts by JonnieCache on Friday February 25, 2011 @07:04AM (#2261887)
Failover in this scenario would require instant, 100% replication of a near-zero latency system.

I'm no expert, but I'm fairly sure having trades 'disappear' or any other loss of consistency would be worse than downtime in this case.

Re:LSE Halts by tallanvor on Friday February 25, 2011 @06:20AM (#2261823)
Unfortunately sometimes fail over isn't a solution. Until they've at least narrowed down the problem, a fail over system may make things worse.

Personally, I'm curious about the testing that was done - given the complaints and problems that have been reported so far, did they even try to run the new and old systems simultaneously for any period of time to verify that the various components could all keep up properly? Or at the very least, did they try to test the system at full load for a decent period of time?

Re:LSE Halts by drstrangevibes on Friday February 25, 2011 @04:53AM (#2261694)
this is the official place for updates, although its not very responsive at the moment,probably due to load http://liveservice.londonstockexchangegroup.com/en/incidents...
Re:LSE Halts by pnathan on Friday February 25, 2011 @10:28AM (#2262443)
It's been archived - here's the link to this incident in the archive:

http://liveservice.londonstockexchangegroup.com/en/incidents...

Re:LSE Halts by JonnieCache on Friday February 25, 2011 @05:25AM (#2261753)
I wonder if it's HFT related? A group of rogue algorithms gone exponential? There are certainly people worried about that...