Put very bluntly, in high frequency domains you compete with fast people (and they have resources) but in mid- or low frequency you compete with smart people. The problem is much harder but therefore it's also much easier :)
This is not to say that I'm a profitable quant (because I'm not) but I think my chances are much higher in the are I chose.
The question that seems interesting: is it possible to guess the price movement for some t+delta moment, where the delta could be for instance 0.1 sec? Or it would be completely unpredictable...
How about doing this not with 1000 equities but with say SPY options, or any other high volume options on cboe ? As long as your derivatives are ATM or say max 2 hops from ATM on either side & open interest is high, it would work ok.
I've only had a glance, but it looks very well done.
Meta note: I especially like how the code was intertwined with descriptive text like an R Knitr file. It makes it easy to follow allow and verify that what the author says he's doing, is actually what he's doing:)
A few issues about using this in production, none of which are intended to slight article's author or his work.
1) The biggest issue for me has always been speed. In this instance he's using one symbol, imagine trying to do this against 1000 symbols with the real depth of market and not just the precanned market data he's using. its alot more data to parse and classify. You can start to see why many HFT systems are more of an IT and programming endeavor than a quantitative one, not that the quantitative portion isn't important:)
2) Just being able to detect which way the stock will tick doesn't really help as much as you'd think.
Assume that you can correctly identify the direction of the next tick 100% of the time. To make money off of this information you need to:
- be able to do this classification and send the order to market faster than it takes to receive the next tick(n+1), a difficult task for most stocks that trade in major US markets.
- Get to the top of the order book, again a difficult task as the bid/ask spread is already tight, and sometimes at its penny limit.
- Get someone to fill your order, again a factor of being at the top of the order book,
- Identify if the tick n+2 is going in the same or opposite direction. if you guess wrong, then you lose. If you guess right you still need to be able to exit your position.
As always, if you are capable of doing this kind of work and able to work in Canada. I'd love to chat with you!! Heck even if you are interested in machine learning and the markets, I'll make time to chat.