News for hackers. Stuff that matters.
Re:Ask HN: Is it feasible to do high-frequency trading as an... by lrm242 on Thursday July 15, 2010 @09:18AM (#1517671)
True high-frequency is very hard. Most of these guys got started in the 1999/2000 time frame and didn't have to deal with many of the start-up issues that new entrants face. For example, just consider the data you need to trade that quickly. Not only is the real-time feed expensive, but if you want to source the data from the execution venue then you have a lot of code to write. The amount of data is also quite large. For example, the US equities market executes approximately 45 million trades per day and there are about 700 million quotes per day, not including the non-top of book quote activity in the various ECNs. I collect between 2 and 4 GB of data per day, if you were to go direct to the execution venues you'd be looking at >50 GB per day, probably.

So, anyway, the "true" high-frequency game is very tough today because we're already 10+ years into it. The markets have changed and the edge has gotten smaller, but there is still plently for a lowly individual automated trader to scratch away at.

Instead of looking for millisecond opportunities, look for second or minute opportunities. Go where the big guys can't because there isn't enough capacity. Can you find an edge that, on average, keeps you in a trade 30 seconds, for example? There's still plenty of alpha left, just don't step onto their playground and expect to get onto the swing set.

A few practical notes:

* You need to look into unbundled or cost-plus commission structures. These fee structures charge a per share commission and pass through all fees and rebates from the executing venue. This is required to do any sort of size with reasonable cost.

* Most "retail" brokers are not sufficient for any sort of high-volume algorithmic trading. Interactive Brokers is barely ok if you are in any way interested in limit order trading because they have fairly large cancel fees for direct routed orders. If you're model doesn't rely heavily on strictly offering liquidity or you're ok with letting IB route your order then IB is ok and offers an unbundled commission structure. Lightspeed Trading and Lime Brokerage are two that cater to active individual and institutions.

* Data storage is a big deal. Effective storage of regular (evenly spaced) and irregular time series will require you to engineer something. There are commercial solutions, but you can't afford them. When you're dealing with high-volume intraday trading this is one of the first issues you'll face. How do you store, query, and manipulate data that includes 45 million new rows per day? The relational DBs fall apart pretty quickly and even if they didn't they won't give you the time series operations you need/want.

* Data feeds are expensive, but required. Look at DTN NxCore. It is a full market feed that will give you the best you can get w/o going direct to the exchange. Some brokers will give you access to raw exchange feeds, but you'll need to engineer feed handlers and a ticker plant for them. This is a non-trivial task, but not impossible. Once you've done that, you'll need to figure out how to get all that lovely data off your co-located server and back to your home base for analysis. Network engineering will be required because your broker doesn't want you pushing 10-20GB per day through their network connection, so you'll need a circuit from an on-premise carrier.

* The banks are players in HFT, but not the original or best. Most of the guys that started it are still independent. Look at GETCO, RGM Advisors, etc.

* Flash orders, what most folks in this thread are refering to when they say the exchange gives the HFT firm a first look, are no more. That edge existed, and I'm sure HFT took advantage, but no HFT firm was built on flash orders. When they started flash orders didn't exist.

It is possible to be a successful, independent, automated trader. It is even possible to do it on a purely intraday, high-volume basis. Don't get caught up in the hype of needing to be high-frequency or not.

---

EDIT: The other thing I forgot to mention is that naked sponsored access is likely going away. This is a near-requirement for "true" HFT. Any future regulation won't affect any of the existing players because they're all grown up now and most have their own broker/dealers. Some form of sponsored access will likely still exist, but pre-trade risk checks will probably be required and will therefore still leave you're broker between you and the market.

Re:Ask HN: Is it feasible to do high-frequency trading as an... by keltex on Thursday July 15, 2010 @11:02AM (#1518009)
NPR's Podcast Planet Money has an interesting show on this subject called "The Million Dollar Microsecond":

http://www.npr.org/blogs/money/2010/06/08/127563433/the-tues...

Re:Ask HN: Is it feasible to do high-frequency trading as an... by anigbrowl on Thursday July 15, 2010 @11:39AM (#1518130)
(very off topic) How do you store, query, and manipulate data that includes 45 million new rows per day?

Perhaps one shouldn't? I've always been a bit fascinated by the analytical tools in brokerage software - with a pretty good understanding of DSP and an appreciation for the fact that asset prices are somewhat periodic, it's hard to overlook the fundamental similarities between stock graphs and audio waveforms. Once you start performing FFTs or wavelet transforms and get a 'feel' for dealing with signals, patterns become very seductive...possibly too seductive: http://en.wikipedia.org/wiki/Pareidolia and http://en.wikipedia.org/wiki/Tetris_effect

Might there be another approach? You wouldn't prepare to go to the store by reviewing and analyzing the 1287 individual footsteps of your previous trip, or try to predict the content of a HN thread by textual analysis of all previous threads. Do we do so at a subconscious level, then? Not really - or rather, our subconscious tends to forget about things as soon as they cease to be important, which in the case of things like walking is a period of seconds or less. Processing large volumes of data is computationally expensive, but it turns out that simple rules can yield results that are both complex and useful, as in flocking and swarming behavior: see http://en.wikipedia.org/wiki/Boids and http://en.wikipedia.org/wiki/Swarm_Intelligence, plus everything from the wisdom of crowds to nonlinear dynamic systems (aka chaotic ones) like Newton's basin or the logistic equation.

I feel there are two other fundamental problems with the massive dataset + analysis approach. One is that you're not working in a closed system, and there's no sensible way to quantify unexpected events. 'Bigcorp CEO in Sex Scandal!' might cause the price of Bigcorp to tank if it's a major distraction or their largest customer base is among rural conservatives. If Bigcorp makes racing cars, it might just be good publicity! Now you can do some kinds of interesting posthoc analysis (eg for news stories that contain a stock symbol, measure the correlation between # of textually similar stories and stock volume/prices using a distributed windowing function) but we're a long way from having a browser plugin that trades based on the contents of your RSS feed.

Another problem is that of feedback. As you've discussed so ably above, people who spot an arbitrage opportunity will mine the hell out of it. And as we all know, traders are extremely subject to herding behavior even though all training suggests they do otherwise. Sure there are systematic contrarians, but I bet that if you just want to do academic analysis you could find a contrarian coefficient and quantify its damping effect on price or volume movements.

So rather than crunching vast quantities of stored data, I wonder if it might be better to treat price movements not as absolutes which you hope will reach a particular ceiling or floor, but as differential vector data with a short half-life. So far AI and modeling approaches seem to have focused on prediction (surprise) and don't perform especially well. I think it would be more interesting to map correlation variations for as large a number of nodes (listed securities) as possible - think how we intuitively appreciate the dynamics of a school of fish when watching a nature documentary, without performing any detailed analysis of individual fish trajectories.

Of course this still involves processing a lot of data, but storing it is less important because are only seeking to become more familiar with high-level behaviors inside that system. There's more to fishing than running trawlers!

Re:Ask HN: Is it feasible to do high-frequency trading as an... by lrm242 on Thursday July 15, 2010 @11:54AM (#1518170)
Good thoughts, and I don't particularly disagree. Regardless of any derivative you might obtain from the data, most folks want to retain high-fidelity historical data for two reasons:

(a) Future analysis techniques are unknown. Today you might be using method X, but tomorrow you might want to try method Y which calls for an entirely different massaging of the raw data.

(b) Backtesting and replay. Simulations and replays of previous trading events are valuable not only for testing a model but also testing the particulars of new "infrastructure" code.

Both require high-fidelity source data. This stuff is so hard to come by and costs so much money that most people want to be safe and save it forever.

To your point on analysis: it is rare that one would directly analyze the source data itself. You'll almost always want to transform the data into something more manageable for model development. The most crude form of this is idea of "bars". Instead of looking at tick data, traders tried to reduce the noise by looking at arbitrary aggregations of that data: 1 minute, 1 hour, 1 day, etc. Technical analysis using moving averages and other indicators are also examples.

Re:Ask HN: Is it feasible to do high-frequency trading as an... by anigbrowl on Thursday July 15, 2010 @12:43PM (#1518305)
Seems like hosting a large hifi dataset in the cloud (eg rolling last 5 years) and charging a small fee to crawl it might be a good opportunity. Or maybe there's no margin in it when people are prepared to pay $$$ as you describe even if they're reinventing the wheel in the process.

Not that I'm a mathematical or economic genius of any kind, but I continue to be surprised at how primitive financial analytics seem. When people do find something interesting (eg Li's Gaussian Copula) they almost invariably make a fetish out of it and hurl themselves off the nearest cliff shortly afterwards. Economics faculties are as much to blame as anyone, I feel.

Re:Ask HN: Is it feasible to do high-frequency trading as an... by wglb on Thursday July 15, 2010 @09:19PM (#1519865)
It is likely that you can't share that tick data--there are redistribution restrictions.
Re:Ask HN: Is it feasible to do high-frequency trading as an... by pbhjpbhj on Friday July 16, 2010 @09:04PM (#1523071)
Stock exchange data is closed? I find that rather strange.
Re:Ask HN: Is it feasible to do high-frequency trading as an... by lrm242 on Friday July 16, 2010 @09:58PM (#1523145)
It's not closed, but to purchase re-distribution rights is very expensive.
Re:Ask HN: Is it feasible to do high-frequency trading as an... by matwood on Thursday July 15, 2010 @05:30PM (#1519279)
Interesting thoughts. I had an acquaintance years ago who did a type of HFT. He built and ran a team of people for 5 years or so before 'retiring' with his FU money and traveling the world.

With respect to your news about a CEO, what he told me is that any stocks that had news or upcoming news were simply pulled off the table from trading. I always thought that there could be some value in parsing and understanding news, but from his standpoint it simply took too long. His other point is why deal with other variables when you don't have to. Find low news stocks and you find something that is more easily predictable.

That leads me to the next point he made, time. I asked him about using exotic analysis techniques and again he said most of their algos were variations of regression testing. Now, he didn't go into how they figured out the variables in their regressions, but he said the reason they stuck with those variations is because they were fast and they allowed them make decisions quickly before the window of profit closed.

Last I heard he retired at 30 traveled the world with his wife got bored and now runs a hedge fund. I don't know if how he was doing it 10 years ago would apply to today.