News for hackers. Stuff that matters.
Re:Stock Price Prediction with Big Data and Machine Learning by chollida1 on Tuesday November 25, 2014 @01:42PM (#8658995)
I read this last night and thought it was a great writeup.

Meta note: I especially like how the code was intertwined with descriptive text like an R Knitr file. It makes it easy to follow allow and verify that what the author says he's doing, is actually what he's doing:)

A few issues about using this in production, none of which are intended to slight article's author or his work.

1) The biggest issue for me has always been speed. In this instance he's using one symbol, imagine trying to do this against 1000 symbols with the real depth of market and not just the precanned market data he's using. its alot more data to parse and classify. You can start to see why many HFT systems are more of an IT and programming endeavor than a quantitative one, not that the quantitative portion isn't important:)

2) Just being able to detect which way the stock will tick doesn't really help as much as you'd think.

Assume that you can correctly identify the direction of the next tick 100% of the time. To make money off of this information you need to:

- be able to do this classification and send the order to market faster than it takes to receive the next tick(n+1), a difficult task for most stocks that trade in major US markets.

- Get to the top of the order book, again a difficult task as the bid/ask spread is already tight, and sometimes at its penny limit.

- Get someone to fill your order, again a factor of being at the top of the order book,

- Identify if the tick n+2 is going in the same or opposite direction. if you guess wrong, then you lose. If you guess right you still need to be able to exit your position.

As always, if you are capable of doing this kind of work and able to work in Canada. I'd love to chat with you!! Heck even if you are interested in machine learning and the markets, I'll make time to chat.

Re:Stock Price Prediction with Big Data and Machine Learning by terranstyler on Tuesday November 25, 2014 @02:43PM (#8659313)
For this very reason I always try to stay in the "mid to low frequency" spectrum.

Put very bluntly, in high frequency domains you compete with fast people (and they have resources) but in mid- or low frequency you compete with smart people. The problem is much harder but therefore it's also much easier :)

This is not to say that I'm a profitable quant (because I'm not) but I think my chances are much higher in the are I chose.

Re:Stock Price Prediction with Big Data and Machine Learning by haddr on Tuesday November 25, 2014 @01:48PM (#8659022)
Exactly, the main problem is that it might be much easier to do such prediction "a posteriori" rather than use such algorithms in the real-time bidding...

The question that seems interesting: is it possible to guess the price movement for some t+delta moment, where the delta could be for instance 0.1 sec? Or it would be completely unpredictable...

Re:Stock Price Prediction with Big Data and Machine Learning by skillachie on Wednesday November 26, 2014 @12:12AM (#8661173)
I would also be very interested in seeing this applied to predictions at t + delta as well. Will probable attempt it over the Christmas break
Re:Stock Price Prediction with Big Data and Machine Learning by haddr on Wednesday November 26, 2014 @07:30PM (#8665379)
Great! let me/us know your results!
Re:Stock Price Prediction with Big Data and Machine Learning by dxbydt on Tuesday November 25, 2014 @03:08PM (#8659421)
>imagine trying to do this against 1000 symbols

How about doing this not with 1000 equities but with say SPY options, or any other high volume options on cboe ? As long as your derivatives are ATM or say max 2 hops from ATM on either side & open interest is high, it would work ok.

Re:Stock Price Prediction with Big Data and Machine Learning by lordnacho on Tuesday November 25, 2014 @02:38PM (#8659289)
You can use the advanced technique to uncover heuristics or simplifications that are faster to trade. Conclusions that take a fine instrument to reach may not need such a fine tool to act upon.

I've only had a glance, but it looks very well done.

Re:Stock Price Prediction with Big Data and Machine Learning by reality on Tuesday November 25, 2014 @02:48PM (#8659338)
Thanks for nice overview ;)