News for hackers. Stuff that matters.
Re:Ask HN: Is it feasible to do high-frequency trading as an... by anigbrowl on Thursday July 15, 2010 @11:39AM (#1518130)
(very off topic) How do you store, query, and manipulate data that includes 45 million new rows per day?

Perhaps one shouldn't? I've always been a bit fascinated by the analytical tools in brokerage software - with a pretty good understanding of DSP and an appreciation for the fact that asset prices are somewhat periodic, it's hard to overlook the fundamental similarities between stock graphs and audio waveforms. Once you start performing FFTs or wavelet transforms and get a 'feel' for dealing with signals, patterns become very seductive...possibly too seductive: http://en.wikipedia.org/wiki/Pareidolia and http://en.wikipedia.org/wiki/Tetris_effect

Might there be another approach? You wouldn't prepare to go to the store by reviewing and analyzing the 1287 individual footsteps of your previous trip, or try to predict the content of a HN thread by textual analysis of all previous threads. Do we do so at a subconscious level, then? Not really - or rather, our subconscious tends to forget about things as soon as they cease to be important, which in the case of things like walking is a period of seconds or less. Processing large volumes of data is computationally expensive, but it turns out that simple rules can yield results that are both complex and useful, as in flocking and swarming behavior: see http://en.wikipedia.org/wiki/Boids and http://en.wikipedia.org/wiki/Swarm_Intelligence, plus everything from the wisdom of crowds to nonlinear dynamic systems (aka chaotic ones) like Newton's basin or the logistic equation.

I feel there are two other fundamental problems with the massive dataset + analysis approach. One is that you're not working in a closed system, and there's no sensible way to quantify unexpected events. 'Bigcorp CEO in Sex Scandal!' might cause the price of Bigcorp to tank if it's a major distraction or their largest customer base is among rural conservatives. If Bigcorp makes racing cars, it might just be good publicity! Now you can do some kinds of interesting posthoc analysis (eg for news stories that contain a stock symbol, measure the correlation between # of textually similar stories and stock volume/prices using a distributed windowing function) but we're a long way from having a browser plugin that trades based on the contents of your RSS feed.

Another problem is that of feedback. As you've discussed so ably above, people who spot an arbitrage opportunity will mine the hell out of it. And as we all know, traders are extremely subject to herding behavior even though all training suggests they do otherwise. Sure there are systematic contrarians, but I bet that if you just want to do academic analysis you could find a contrarian coefficient and quantify its damping effect on price or volume movements.

So rather than crunching vast quantities of stored data, I wonder if it might be better to treat price movements not as absolutes which you hope will reach a particular ceiling or floor, but as differential vector data with a short half-life. So far AI and modeling approaches seem to have focused on prediction (surprise) and don't perform especially well. I think it would be more interesting to map correlation variations for as large a number of nodes (listed securities) as possible - think how we intuitively appreciate the dynamics of a school of fish when watching a nature documentary, without performing any detailed analysis of individual fish trajectories.

Of course this still involves processing a lot of data, but storing it is less important because are only seeking to become more familiar with high-level behaviors inside that system. There's more to fishing than running trawlers!

Re:Ask HN: Is it feasible to do high-frequency trading as an... by lrm242 on Thursday July 15, 2010 @11:54AM (#1518170)
Good thoughts, and I don't particularly disagree. Regardless of any derivative you might obtain from the data, most folks want to retain high-fidelity historical data for two reasons:

(a) Future analysis techniques are unknown. Today you might be using method X, but tomorrow you might want to try method Y which calls for an entirely different massaging of the raw data.

(b) Backtesting and replay. Simulations and replays of previous trading events are valuable not only for testing a model but also testing the particulars of new "infrastructure" code.

Both require high-fidelity source data. This stuff is so hard to come by and costs so much money that most people want to be safe and save it forever.

To your point on analysis: it is rare that one would directly analyze the source data itself. You'll almost always want to transform the data into something more manageable for model development. The most crude form of this is idea of "bars". Instead of looking at tick data, traders tried to reduce the noise by looking at arbitrary aggregations of that data: 1 minute, 1 hour, 1 day, etc. Technical analysis using moving averages and other indicators are also examples.

Re:Ask HN: Is it feasible to do high-frequency trading as an... by anigbrowl on Thursday July 15, 2010 @12:43PM (#1518305)
Seems like hosting a large hifi dataset in the cloud (eg rolling last 5 years) and charging a small fee to crawl it might be a good opportunity. Or maybe there's no margin in it when people are prepared to pay $$$ as you describe even if they're reinventing the wheel in the process.

Not that I'm a mathematical or economic genius of any kind, but I continue to be surprised at how primitive financial analytics seem. When people do find something interesting (eg Li's Gaussian Copula) they almost invariably make a fetish out of it and hurl themselves off the nearest cliff shortly afterwards. Economics faculties are as much to blame as anyone, I feel.

Re:Ask HN: Is it feasible to do high-frequency trading as an... by wglb on Thursday July 15, 2010 @09:19PM (#1519865)
It is likely that you can't share that tick data--there are redistribution restrictions.
Re:Ask HN: Is it feasible to do high-frequency trading as an... by pbhjpbhj on Friday July 16, 2010 @09:04PM (#1523071)
Stock exchange data is closed? I find that rather strange.
Re:Ask HN: Is it feasible to do high-frequency trading as an... by lrm242 on Friday July 16, 2010 @09:58PM (#1523145)
It's not closed, but to purchase re-distribution rights is very expensive.
Re:Ask HN: Is it feasible to do high-frequency trading as an... by matwood on Thursday July 15, 2010 @05:30PM (#1519279)
Interesting thoughts. I had an acquaintance years ago who did a type of HFT. He built and ran a team of people for 5 years or so before 'retiring' with his FU money and traveling the world.

With respect to your news about a CEO, what he told me is that any stocks that had news or upcoming news were simply pulled off the table from trading. I always thought that there could be some value in parsing and understanding news, but from his standpoint it simply took too long. His other point is why deal with other variables when you don't have to. Find low news stocks and you find something that is more easily predictable.

That leads me to the next point he made, time. I asked him about using exotic analysis techniques and again he said most of their algos were variations of regression testing. Now, he didn't go into how they figured out the variables in their regressions, but he said the reason they stuck with those variations is because they were fast and they allowed them make decisions quickly before the window of profit closed.

Last I heard he retired at 30 traveled the world with his wife got bored and now runs a hedge fund. I don't know if how he was doing it 10 years ago would apply to today.