You make a good point. I've also gotten into trading because I enjoy the algorithmic and mathematical aspects, and I would love to share more of what has been working for me and write extensively about it. And there are probably more people like that out there. However, trading has such a bad reputation and uncertain future that I am not sure that's a good career move. I'm torn.

You're right that there are probably some gems and people writing up good posts and articles. However, 99% of what comes to my inbox, which is certain newsletters and arXiv subscriptions, is clearly BS. I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post. In trading the opposite seems to be true. Seeing a good trading paper on arXiv is incredibly rare. I would even go as far as saying that reddit is a significantly better source of information than arXiv for this field.

Have you tried quantpedia? https://quantpedia.com/

It's expensive but I find it a really good source for ideas.

So how should we evaluate the quality of a paper on trading AI? I mean the authors might not have access to real data, but their ideas might still be good.

There are some ML problems where it is fundamentally impossible to use historical data to make accurate forward looking predictions as its not IID. These fields require you very carefully capture data on sub-optimal choices. In the case of trading this means making explicitly bad trading decisions some portion of the time, and teams that have done this at any scale are unlikely to share the data.

In the case of trading, any paper not tackling these issues head on is not likely to be useful.

I don't get it - if you have accurate historical data, how is this different from having access to current real-time data? Why can't you pretend you live 20 years in the past and use the data you have as if it were real-time?

Data distribution shift. The market changes over time and your current data does not come from the same distribution as old data. That limits the amount of data you can use for training and testing. You need to be very careful not to overfit. That's especially true for something like daily or hourly data - there isn't much data to begin with and you won't have much left if you look at only a few weeks or months. Market data already has a low signal/noise ratio to begin with, so you need a good chunk of data to learn from.

As you go to shorter time scales you get more usable data, but then you also need to deal with other issues such as latencies/jitter, market impact, complex order types, order book queues, etc. It becomes a different game.

For 1 because your trading existence in that universe would change the future which you can't account for. Your activity influences decisions of other HFTs in real time whereas with a static history you're claiming to be able to trade without perturbing the markets.

Fundamentally, the issue is that in real time you may not be able to make the trade that your algorithm chose. You could get close if you had the actual book prices at any given time, but even then, you might lose to someone who is 1 millisecond faster. So no, backtesting can simulate reality. Interactive Brokers offers a simulated account where you can practice "live" trading, although it's still not the same, since there's no money involved. But if I see a paper tested on an IB simulated account I'll be very interested, and it'll be too late already.

Good point, but I was mainly thinking about making a single good decision to multiply your investment, not HFT. Like identifying that it was a good idea to invest in Tesla stock 7 years ago.

Papers on trading models that show feature importance (rather than backtest results) are more valuable.

>I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post.

You should really google up something called the Gell-Mann amnesia effect. 99.9% of everything is shit. Including biology, CS, ML and especially "AI."

Of course trading papers are even more universally shit, but once in a while someone publishes a non obvious to me risk factor.

Thanks for the Gell-Mann amnesia effect.

Would you kindly give me a fair idea as to what’s a good amount of money to be made in this field?

As with most industries, it depends. But junior people typically make in the 200-500K range. Then as you gain experience, develop your own ideas/strategies and are able to manage risk appropriately, the sky is the limit. The closer you are to managing money that's being invested the more you make. If you can run a 1.5 - 2 Sharpe strategy and never dip below a ~5% drawdown, i.e. probably have substantial positive skew in returns, you can make in the millions or tens of millions at the right fund. Note that as the OP correctly alluded to, this is much more difficult to do live than in a backtest.

Is 200-500k still true? It used to be, but I think it has decreased significantly over the last decade. I'd say most junior people in this field are making about the same or less than software engineers these days.

But like you said, the range here is incredibly wide and largely depends on how well your strategies do and if you have your own desk/fund.

I don’t think it is. I worked for one of the major HFTs and new graduates earned far less than that. In addition the churn rate was high - I’d say most new graduate hires didn’t last more than 2 years and what you learned in those 2 years was often not much use in terms of experience useful for other career paths.

Bonus distribution was reverse exponential. Like traditional consultancy partnerships, a small few of the old hands made serious money but those at the “bottom” made ok money but after a few years their FAANG based contemporaries were doing better. Advancing up the ranks was not guaranteed even if you survived the frequent blood lettings.

I wonder if this isn't the cause of lack of progress in science. - Why create wealth for all when you can acquire currency for yourself by managing other people's money?

It seems to me like those tragic stories of genuises who died young. What could have been if their ideas had reached the world? But instead of dying the geniuses got sequestered into finance and secrecy, volunteering to make no mark at all on the world of their passing.