Example of a pretty interesting and accessible one - is "101 Formulaic Alphas" [0].
Alpha#33: rank((-1 * ((1 - (open / close))^1)))
rank(open/close - 1)
Martingale is a sure-fire way to lose all your money rather quickly.
> Alpha#90: ((rank((close - ts_max(close, 4.66719)))^Ts_Rank(correlation(IndNeutralize(adv40, IndClass.subindustry), low, 5.38375), 3.21856)) * -1)
I wonder how these magic numbers get picked (4.66719, 5.38375, etc) -- I guess there is some optimization solver which attempts to find the most profitable variables for a given alpha formulation, but isn't this approach also very vulnerable to overfit?
These alphas will likely be only profitable for a short time period as long as the market data distribution (i.e. strategies of other market participants) doesn't change. So you would need to continually optimize and update them.
The way I think about it is that you are essentially finding the right parameters to "exploit" the combination of algorithms of all other participants, where algorithm could also be a human looking at charts and following certain rules, with a lot of random noise from retail traders thrown in.
> (sign(delta(volume, 1)) * (-1 * delta(close, 1)))
That's crazy. Would be interesting to see WTF a "mega-alpha" actually does using these strategies.
Well, that is mostly true. But never discount anything. There are people like me who used to love the data analysis and prediction part in these markets. I got hooked to the markets because of it. I was not interested in making money and naively thought my average pay was good enough. When I first built (or my machine built) a working strategy (in early 2008), live traded/tested it for a couple of months and told few colleagues about the details about the strategy - they did not take me seriously. This was even before I understood NNs or any of scikit-learn tooling. I knew I wanted to get into financial markets - went to a broker to sell the automated strategy and seeking a full time job as an algo-trader - they thought I was trying to scam them even after seeing the contract notes. Plus algotrading had not picked up back then. I found later about such scams. It took me 3 more years and a financial crisis to understand the value of making "much more than enough" money. And retrospectively I know those were just stupid attempts trying to convince others and attempting to give it away.
You're right that there are probably some gems and people writing up good posts and articles. However, 99% of what comes to my inbox, which is certain newsletters and arXiv subscriptions, is clearly BS. I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post. In trading the opposite seems to be true. Seeing a good trading paper on arXiv is incredibly rare. I would even go as far as saying that reddit is a significantly better source of information than arXiv for this field.
It's expensive but I find it a really good source for ideas.
In the case of trading, any paper not tackling these issues head on is not likely to be useful.
As you go to shorter time scales you get more usable data, but then you also need to deal with other issues such as latencies/jitter, market impact, complex order types, order book queues, etc. It becomes a different game.
You should really google up something called the Gell-Mann amnesia effect. 99.9% of everything is shit. Including biology, CS, ML and especially "AI."
Of course trading papers are even more universally shit, but once in a while someone publishes a non obvious to me risk factor.
But like you said, the range here is incredibly wide and largely depends on how well your strategies do and if you have your own desk/fund.
Bonus distribution was reverse exponential. Like traditional consultancy partnerships, a small few of the old hands made serious money but those at the “bottom” made ok money but after a few years their FAANG based contemporaries were doing better. Advancing up the ranks was not guaranteed even if you survived the frequent blood lettings.
It seems to me like those tragic stories of genuises who died young. What could have been if their ideas had reached the world? But instead of dying the geniuses got sequestered into finance and secrecy, volunteering to make no mark at all on the world of their passing.
If they haven't tested this in actual trades and measured results, it's probably worthless. Even backtested strategies at actual firms observe decays (or don't work) when they get put live. And those are places where they invest in (and are incentivized to get right!) backtesting methodology.
I think this is a little unfair. I've seen high-quality papers from phD students who then get hired by financial firms and were apparently very successful. Every good real-world AI system requires both good engineering and good science and it's disingenuous to suggest that all science that isn't actively being applied yet is BS.
As a side note, what this specific paper here did is neither novel not innovative, so it's very fair to criticize it. A3C is 4 years old, and they just take it and run it on some data. It's like downloading a convnet and running it on MNIST. There have been hundreds of papers on RL + Trading. I see them in my arXiv emails every other day and they all do the same thing.
> This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system
> Whenever you see a paper ending with a backtest you can already assume it's BS.
https://www.johndcook.com/blog/2011/06/21/how-to-fit-an-elep...
Edit down voters care to elaborate?
I know this, and I ran a company where people should know this, but so many people are so easily swayed by "authority"
like, so and so made trading programs for Investment Bank Co 20 years ago so you know their trading algorithm has to have merit
uh no, they are not retired, they are broke and can't even fund $10k into a trading account to try it
at this point all I would say is just smile and nod.
Unfortunately it seem like most ML people are not really interested in trading, perhaps because it has such a bad reputation (which is IMO unjustified) - so they work on games instead :)
You couldn't be more wrong on this. Stock market trading has the lowest barrier to entry of any endeavor. All you need is $1000 and Robinhood account, which you can open one on your phone in 5 min or less.
I've been following HN for a while, every time someone comes up with a trading algo or posts a link to algo, there's were hundreds of upvotes, lots of comments.
Quandl has many free, or low cost stock market/commodity datasets.
I'm not sure what you mean by a "simulator". One of the greatest challenge applying RL to stock mkt is precisely that the market itself is not a MDP.
There is a good reason trading firms pay a lot of money (sometimes millions) for fine-grained historical data from exchanges. It's not only about speed. For interesting experiments you IMO need L2 or L3 order book data, ideally somewhere on second or sub-second scales. That's not HFT (which is nano and micros), but somewhere in the "middle" - it's a different world than what you are talking about.
By simulators he means market simulators for L2/L3 data with a matching engine, latencies, queue positions, jitter, complex order types, etc. You can't simulate other market participants (at least not fully, but there are techniques to even estimate this based on live trading feedback), but there are still many things left that you can simulate in a realistic way during training and backtesting. Trading companies typically have their own high-performance simulators built in house. Some of these are incredibly complex. Good simulators can give you a huge edge and are absolutely necessary.
"outliers and events outside of the data, news" : these are precisely the stuff your models need to learn, and the fact that you consider them noise tells me most folks have no clue how to predict these "noise".
I've long understood that this was true. It makes intuitive sense.
But are there any cases where it is not true?
Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor?
Or does it always reduce down to "I win only because you lose"?
Then there is the cultural aspect. People who are working in trading are just not used to sharing openly. They don't write online, or anywhere. They are not even allowed to write due to their employers. And people who work in academia are naturally not working on "production" systems - their only job is to write, not to build. So you almost never see people in the intersection of: writes-online & understands-trading & is-not-in-academia
But that's an exception rather than a rule
The real value though of publishing a trading strategy is in signaling to future employers.
Ultimately, money is made by the ability to come up with new strategies. Any single strategy is only going to live for so long before it dies and it is no longer profitable.
Opening up one's secrets of trading seems to only make sense if one has found deeper, more effective secrets, so that the old crop is not going to be seriously competitive, but a bit of good PR would come in handy.
Something like:
| | Play | Don't Play |
|------------+------+------------|
| Play | -5 | +10 |
| Don't Play | -10 | 0 |> In game theory and economic theory, a zero-sum game is a mathematical representation of a situation in which each participant's gain or loss of utility is exactly balanced by the losses or gains of the utility of the other participants.
[1] You can change the +10 to +9 if you want to make it the absolute highest total payout.
For example, a gold miner may sell gold futures to guarantee that he won't go out of business once the construction of the new gold mine is complete. There are many other examples.
I find that in reality opportunity cost rarely matters.
It has absolutely nothing to do with opportunity cost, or whether you, personally, are a "loser". But if you're a "winner", someone else must be the "loser".
Let's say I am a market maker offering to buy Apple shares at $99 and sell them at $100. Let's take an ex-Apple employee who owns some shares. He just had a family emergency and wants to liquidate his shares to get cash, and he needs it quickly. He doesn't care about paying a few dollars extra in exchange for a quick trade because he needs to pay a bill tomorrow. I buy his shares for $99. He is happy because he immediately got his cash.
On the other side, there is a a retail investor doing long-term investment and wants to add Apple to their portfolio. They also don't care about a few cents because they're holding the stock for a decade and love the new CEO. They buy my Apple shares from me for 100.0. They are happy because I can guarantee them a stable price for a decent number of shares.
All participants are happy. I just made $1 from the spread for providing liquidity, the investor got the long-term investment they wanted, and the ex-Apple employee got his cash.
Sure, both sides of the market could have made more optimal trades if they had put in more effort and "optimized" their trades with algos and somehow skipped the middle-man, but they would've sacrificed convenience and time, which may be worth more to them than the little bit of extra $ they paid. Aren't we all winners?
When you go buy bananas in your grocery store you also don't complain about them taking a cut for providing liquidity. You don't say the farmer has "lost" money because the consumer paid more than what the farmer originally sold for to the grocery store. The farmer is happy because otherwise he may not have traded at all or his bananas may have gone bad (= needs to trade quickly). This is no different.
In addition, companies produce things, some of that wealth gets returned to the investors through dividends, interest (eg on bonds) and buy backs (in my example, let's say each unit generates $1 in dividend, now total wealth is $40(!) while starting at $20, including $20 cash (starting from $10) and $20 worth of units (starting from $10)).
In fact, we see this growth everywhere around us as both the amount of people and the amount of goods and services per person is increasing!
It is a zero sum game. Nobody is producing anything, therefore for one to win another must lose.
Many HFT jump out when things get volatile, when liquidity is actually required.
Ultimately HFT is doing nothing of societal value, the race down to zero is never-ending and we are wasting huge amounts of resources on a totally pointless march towards zero. Exchanges should introduce random delays to allow market participants who really want to hedge / buy / sell, then we can shift some of the resources to the real world. The costs required to compete at the lowest latencies are large, and forcing small/medium players out the game, as the investment cost is large, which is also bad.
The system is hugely inefficient. The costs as latencies get lower are ever higher, for an extremely similar end result. The law of diminishing returns.
> Many HFT jump out when things get volatile, when liquidity is actually required.
Do you have a citation on that? If you look the preliminary Q1 results of Virtu Financial [0] (only publicly traded HFT) they seem to be doing more trading than ever in these volatile markets.
> Ultimately HFT is doing nothing of societal value, the race down to zero is never-ending and we are wasting huge amounts of resources on a totally pointless march towards zero.
HFT is a mature industry. Latencies have mostly stabilized, and profitability is way down in the last few years. Many firms are merging/consolidating. So in the past few years society is actually spending fewer resources - both financially and from a human capital standpoint on HFT than it did in the past.
> Exchanges should introduce random delays to allow market participants who really want to hedge / buy / sell, then we can shift some of the resources to the real world.
IEX is doing something relatively similar to that for a few years now. They have ~3% of US equities market share. People have the option of trading there but they mostly choose not to.
> The system is hugely inefficient. The costs as latencies get lower are ever higher, for an extremely similar end result. The law of diminishing returns.
Due to consolidation, costs are actually decreasing. Could it be that the market is... working?
[0] - https://ir.virtu.com/press-releases/press-release-details/20...
Similar story from Flow Traders:
https://www.flowtraders.com/sites/flow-traders/files/quarter...
So what seemed like a quick drive by wasn't actually correct.
You said that HFTs jump out when things get volatile. But then I show you evidence that they trade more in high volatility. Do you not see the contradiction?
> HFT is hugely expensive to maintain. Staffing and equipment costs are massive.
HFT firm equipment costs are a few racks of high end servers and some expensive networking equipment, and a few dozen to few hundred highly paid engineers. Before HFT there were literally _thousands_ of human traders working at different banks doing the same work by hand. Sure in an absolute sense HFT is expensive, but relative to the alternative it is cheap.
> HFT choose not to trade on exchanges with delays because they want to exploit latency advantages. That's why govt should regulate this on all exchanges to just wipe it out.
If HFT is so bad, then the non-HFT market participants should choose to trade at the delay exchanges that make it harder for HFTs. But they choose not to, generally, because the liquidity - and thus trading costs are higher than trading at the non-delay exchanges.
> Consolidation is not good because it will collapse into a monopoly. Which will require regulation also, because markets do not "work" on their own.
First you say that HFT is bad because costs are high, but now you say that falling costs are bad because it will lead to monopoly - do you see the contradiction? For what its worth, Although there are fewer firms there is still a lot of competition among the remaining firms.
We should distinguish between volatility at different timescales. An HFT might well be very active over a volatile month, but may still turn off over a very volatile second. I've always assumed that the "HFTs jump out during volatility" complaint was about the latter; it means that when some shocking news hits the market, the HFT firms providing most of the liquidity pull it, and so the manually-entered market orders wanging around end up moving the market further, exacerbating the volatility. Maybe that's not what people are actually complaining about, though.
> HFT firm equipment costs are a few racks of high end servers and some expensive networking equipment
And FPGAs. FPGA development is really not cheap.
Latency-arb is a small part of HFT, if you consider strategies that provide liquidity to be "good HFT" and strategies that take liquidity (via latency arb or other arbs) to be "bad HFT" then those strategies are largely executed by the same market participants.
> Many HFT jump out when things get volatile, when liquidity is actually required.
This feels almost like a "no true Scotsman" situation. Why is liquidity not "actually required" when volatility is low? Is it a moral obligation for any trader to catch a falling knife? I see this condition of "when liquidity is actually required", but I never understood why there was such a strong feeling for it. Why do you believe this?
> Ultimately HFT is doing nothing of societal value, the race down to zero is never-ending and we are wasting huge amounts of resources on a totally pointless march towards zero.
I don't know, I could probably take a similar view of so many jobs in tech. What does society really get from Snapchat, what do they get from HQ Trivia, what do they get from people making powerpoint presentations with arrows that point to synergies. What's the point of any job with some amount of abstraction?
> Exchanges should introduce random delays to allow market participants who really want to hedge / buy / sell, then we can shift some of the resources to the real world.
Why?
> The system is hugely inefficient
Do you know how efficient the system was before HFT started up? And, do you know how many people were working in trading before, and how many are, for a similar fraction of stock volume?
> The law of diminishing returns.
OK.
Again this weirdly mixes HFT with electronic automated trading, which I really don't think anyone in the domain would readily mix.
HFT by arbing over latency is entirely different to the automation of boring trader tasks that see less people employed to do the same thing in the front office.
I can't continue this more, it's just blind allegiance from people who are clearly not in the domain.
HFT != electronic trading
HFT is also not equivalent to arbing over latency.
For example, an HFT trader make pennies from each trade by exploiting tiny price inefficiencies. He essentially takes money from a "stupid" retail investor who does not know how to optimize his trades. However, the retail investor may not actually care about optimizing trades and just wants to liquidate assets or make a long-term (10+ years) bet. He is totally fine with throwing away a few dollars because optimizing his trades through complex algorithms would be too much work. Here, both parties win, the HFT trades gets paid because he provides convenience, or liquidity, to the retail trader. The same would apply to any human market maker, it doesn't have to be HFT.
And yes, HFT liquidity may disappear during HUGE market movements due to risk, but it doesn't disappear as long as both parties get what they want and the risk is manageable, which is "most of the time". Of course, HFT has other issues such as the race to zero and unfair advantages for a few central players, and I don't want to defend HFT. But saying that "it's all zero sum" is not correct.
An analogy is your nearest grocery store. They're a market maker because they buy from the manufacturer and sell to the consumer and profit from the spread. Do you also argue that these are all zero-sum and we should cut them all out and connect all consumers and farmers directly? And their liquidity also disappears when black swans (corona) happens :)
Melon Usk (say) wants to make cars, but he can’t pay for the factory himself, so he forms a company, sells shares in it, and uses the proceeds to build a factory. Now he and his shareholders can make cars, so the shares are worth more.
Who lost money?
Now, since you appear to know about these things, among all the available papers/article/blogposts/books is there any that you would recommend as being less wrong than the rest? For example, a while ago I read this book [1], and it didn't seem so bad, but I'm not in the industry. Can you recommend anything, even with caveats?
[1] https://www.amazon.com/gp/product/B00BZ9WAVW/ref=dbs_a_def_r...
[0] is okay. I disagree with a lot in there, but it's pretty well written and one of the better books on the subject. [1] Is very old, but it's one of my favorites. It's very mathematical. The ideas still apply today. [2] Is a good introduction overview
[0] https://www.amazon.com/Advances-Financial-Machine-Learning-M...
[1] https://www.amazon.com/Introduction-High-Frequency-Finance-R...
[2] https://www.amazon.com/Trading-Exchanges-Microstructure-Prac...
As a student who is looking to get started with trading and enjoys the mathematical/analysis part of it, do you have advice of where to begin? I find very few resources in this area and its very hard to get on this career path - my experience is on the ML side if things and I want to transition into trading. Any advice will be really helpful - thanks!
https://news.ycombinator.com/item?id=16929156
There are a few different types of roles in the quant world, and number of different types of funds:
alph/signal research: apply quantitative methods to come up with profitable trading ideas and strategies. This is kind of like "Data Science" coming from tech - finding the insight in the data
quant development: build the infrastructure for the data and strategies. This is kind of like "Data Engineering" coming from tech - a lot of ETL and general development work.
portfolio analytics/execution: figure out how to combine different alpha ideals into a portfolio that can be traded. Involves trading and monitoring the live portfolios.
risk management: Thinks of all the possible "risks" the portfolio can be exposed to and ensure they're properly addressed/hedged/accounted for.
This is a broad generalization which can vary greatly from place to place. Typically the smaller funds will have more blurred lines and lots of roles that involve doing multiple of the listed above. At the larger funds, the roles will typically be more well defined and segmented.
Lot's of quant funds are happy to hire people with no finance/trading background if they're strong enough in other key areas. A lot of the "finance" specific stuff can be picked up on the job. Also ML is quite in demand right now.
To front run someone’s order you need to have advance information of their order s? Normally this means the front runner is operating as a broker. I can’t imagine any HFTs using other HFTs as brokers to forward their orders to the exchange?
I’m sure that there is market microstructure stuff/front running practicalities that would make this harder than it sounds but still you wouldn’t completely be in the dark.
Front running has a specific definition in terms of market regulation (see my other comment) and what you describe is not front-running.
Front-running is when someone with a fiduciary duty - typically a broker or dealer - takes an order from a client and then trades on their own book BEFORE executing the client's order knowing the effect of the clients order on the market and knowing that they can exploit this effect for their own benefit.
I know of no HFTs which have such a relationship with rival HFTs and can't even imagine such a relationship existing never mind it being a frequent cause of why strategies perform poorly for HFTs.
Front-running hasn't been a feature of markets for decades at least. Any sniff of front-running would have the SEC or CFTC fine your company into oblivion and possibly result in jail time or at the very least lifetime bans from the financial industry.
Being faster than someone else isn't "front running" them nor is spotting patterns in other participant's behaviour and exploiting those patterns. The definition of "front running" is reasonably specific: https://en.wikipedia.org/wiki/Front_running
Not only is it (very!) easy to overfit backtests (especially with so little data they are using here), but backtests are nothing like the real world. In the real world there are HFT traders front-running you, latency, jitter, fees, hidden order types, slippage, and a lot of other complexities that don't fit into a short HN post. Whenever you see a paper ending with a backtest you can already assume it's BS.
It's similar to training a robot in an extremely simplified 2D simulation environment without physics or other interactions, and then claiming one has built a real robot. A mistake many people make is believing that trading is all about AI. But in reality, the model often matters less than infrastructure/latency/system/data issues.
In addition to that, people who are actually "good" at trading don't publish papers, they silently make money. Papers are typically published by academics or students who have never built anything profitable but would like to put a paper on their resume. I have yet to see a single good academic paper about trading.
[0] https://www.tradientblog.com/2019/11/lessons-learned-buildin...