Although I don't necessarily agree with all the points in this article, it just reminds me what Poincaré said:
`You ask me to predict for you the phenomena about to happen. If, unluckily, I knew the laws of these phenomena I could make the prediction only by inextricable calculations and would have to renounce attempting to answer you; but as I have the good fortune not to know them, I will answer you at once. And what is most surprising, my answer will be right.' -- Poincaré, H. (1913) The Foundations of Science. New York, The Science Press. p. 396.
That said, quant researchers typically understand how the market works. They are just able to quickly excel without a background in it.
You're used to see the techniques you work with capture signal because there isn't an army of PhDs in math, physics, and computer science working around the clock to trade any signal out of that data.
In the end, it doesn't even matter if you're the best statistician in the world: whatever signal you detect may simply not be worth the effort you put into detecting it.
Could it be that by looking only at the prices timeseries we are not looking at the actual information but only at the output of a irreversible function and that for effectively predicting the prices we need a model that captures what actually happens in the real world?
It's fairly true that (at least these days) you're not going to identify a signal just by looking at a timeseries of prices, no matter how granular your dataset is (up to and including tick data). There are pockets of repeating patterns but those are vanishingly small and fleeting; the prices themselves may as well be stochastic.
Essentially all funds are empowered with significant amount of data, and the prices themselves are just used for backtesting and sanity checking. It's the source of truth, but it's not the way in which new insights are identified. The signal comes from other types of data that is far more reversible.
The article cited in the OP says:
""" Our historical dataset contains 5 minute mid-prices for 43 CME listed commodity and FX futures from March 31st 1991 to September 30th, 2014. We use the most recent fifteen years of data because the previous period is less liquid for some of the symbols, resulting in long sections of 5 minute candles with no price movement. Each feature is normalized by subtracting the mean and dividing by the standard deviation. The training set consists of 25,000 consecutive observations and the test set consists of the next 12,500 observations. """
(excerpt from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2756331)
That sounds to me as if the only data they trained the model are a bunch of prices.
I'm I reading it correctly?
We tried sentiment analysis in uni a few years ago and had no good results:
The idea was essentially: news says: 'stock A is great' -> it goes up shortly thereafter
We tested our algorithms against classifying Amazon reviews & Tweets by sentiment. Those are filled with sentiment and easy to detect if it is a 5 star review or a 1 star review. The news articles we parsed all had near neutral sentiment. We ended up building a classifier that could detect the news category of an article quite easily instead.
My initial idea was sparked by the Gulf spill and the subsequent dip in BP, I wanted to detect and capitalise big events like that, but the news sources we parsed always seemed to significantly lag behind the stock movement, too.
I scraped the above sources over a full year (2015) and then had the data annotated on positive, negative and neutral sentiment.
The problem with labeling sentiment data is that there might not be a single 'true' label due to varying interpretations and ambiguity. So at best you'll get to 80-85% accuracy there. The less formal (News > Reddit/Forum > IRC), the lower your accuracy due to lack of context.
I Then matched the annotated sentiment to market data and did some causality analysis. What I found is that interestingly, you can't just say positive news = price/volume goes up. It is way more fine grained than that. For example negative Reddit sentiment leads price movements, but price movements lead positive sentiment. For news its the reverse.
All in all I didn't incorporate this into any trading strategies, but found it interesting to see the differences between online sentiment channels.
Need to measure this in terms of average volume of news for all tracked market participants, and relative to the normal volume of news for that stock.
That way, if the return has recently gone up and there is substantial volume increase its probably because of positive news.
In other words, Rentec is not just pointing machine learning models at data, they're investing in a very robust data processing pipeline. Everything before the analysis is just as important as the analysis at funds like theirs.
While Jim Simons is a mathematician and Rentec clearly has hired many brilliant people with PhDs, it's maybe worth mentioning the actual mathematics being used in their work isn't super high level difficult, impossible or secretive. Many of the PhD's working there do not have a PhD in math, but rather something like Physics, so I would say if you are familiar with graduate level math courses you can understand the math needed for this type of work. Math isn't where their edge comes from. Also Medallion is 30 years old, their early work in the mid 80s was done on computers with less processing power than your phone, "Machine Learning" as the term is being used lately or access to supercomputing hardware no one else knows about is also not where their edge came from.
They need smart people, but hiring the smartest people and having the most sophisticated models won't do you any good if you can't acquire high signal data, can't clean that data properly and can't rapidly backtest. And if you can't do any of that, adding more data is just going to add more noise.
_____________________
By the way for anyone reading this, Rentec is hiring, need to know C/C++ and a "basic knowledge of statistics" is preferred but not required.
Yes.
> Is their advantage in finding signals in data that other people overlook, or finding new data sources?
Yes.
I won't go into any particular detail, but there is a lot of signal in the market for those who are imaginative. The obvious and low hanging fruit is long gone, but there are still many places that offer an edge.
The industry is also extremely secretive (necessarily so). You'll hardly ever find a good treatise on finding novel signals, but there are tutorials on algo trading in general with examples of production strategies that used to work which have been, as you say, traded away. For that purpose I'd recommend you start here: http://www.decal.org/file/2945
ML can be very useful if you have some signal or if you have a model.
Most machine learning techniques focus on problems where the signal is very strong, but the structure is very complex. For instance, take the problem of recognizing whether a picture is a picture of a bird. A human will do well on this task, which shows that there is very little intrinsic noise. However, the correlation of any given pixel with the class of the image is essentially 0. The "noise" is in discovering the unknown relationship between pixels and class, not in the actual output.
Noise dominates everything you will find in statistical arbitrage. R^2 of 1% are something to write home about. With this amount of noise, it's generally hard to do much better than a linear regression. Any model complexity has to come from integrating over latent parameters or manual feature engineering, the rest will overfit.
I think Geoffrey Hinton said that statistics and machine learning are really the same thing, but since we have two different names for it, we might as well call machine learning everything that focuses on dealing with problems with a complex structure and low noise, and statistics everything that focuses on dealing with problems with a large amount of noise. I like this distinction, and I did end up picking up a lot of statistics working in this field.
I'll regularly get emails from friends who tried some machine learning technique on some dataset and found promising results. As the article points out, these generally don't hold up. Accounting for every source of bias in a backtest is an art. The most common mistake is to assume that you can observe the relative price of two stocks at the close, and trade at that price. Many pairs trading strategies appear to work if you make this assumption (which tends to be the case if all you have are daily bars), but they really do not. Others include: assuming transaction costs will be the same on average (they won't, your strategy likely detects opportunities at time were the spread is very large and prices are bad), assuming index memberships don't change (they do and that creates selection bias), assuming you can short anything (stocks can be hard to short or have high borrowing costs), etc.
In general, statistical arbitrage isn't machine learning bound(1), and it is not a data mining endeavor. Understanding the latent market dynamics you are trying to capitalize on, finding new data feeds that provide valuable information, carefully building out a model to test your hypothesis, deriving a sound trading strategy from that model is how it works.
(1: this isn't always true. For instance, analyzing news with NLP, or using computer vision to estimate crop outputs from satellite imagery can make use of machine learning techniques to yield useful, tradeable signals. My comment mostly focuses on machine learning applied to price information. )