Using purely historical price data it is harrowingly difficult. There are 130 anonymized features, so that's unlikely to be only price data. It could include information on the order book, correlated assets, fundamentals, vectorized/embedded text, etc.

Besides, I bet you can train monkeys to do (slightly) better than blindfolded random throwing. Even with public data (replace satellite images with Youtube mentions, or number of links moving into a company website) it is very possible to do better than average guessing on quite a lot of assets (especially smaller and newer markets).

Most hedge funds, even with specialized expensive non-public data, are not magical unicorns. Their quants really may just run a gradient boosting machine and leave it at that. Some hedge funds even prefer linear methods, because this lowers risk through lower variance. Such models can be beaten by experienced Kagglers for sure. For one, I did.

One thing we need to be clear about is that you're not aiming to be better than average. You're aiming to make a profit. There are probably hundreds of thousands of day traders, there are probably <100 market makers and tradingfirms (far less than that for a some specific products) and you'll probably find 99% of the day traders aren't making systematic profits. There are lots of strategies that are much better than average and still worse than putting your cash in a bank.

You can aim for both. If you just aim for profit, then you can get lucky with just average, or even random, betting. If you find a weighted coinflip (which is not impossible), provided by how many times you can flip that coin, you will see steady systematic profits. Of course, majority of day traders are getting owned by the big players, and they would do better doing more reasoned and long-term investments. Most day traders are not even using predictive models though.

On that point - it's pretty clear that this Kaggle competition is highly likely to result in a decent number of submissions that make more money through luck, than other make through strategy.

None of that information is non-public (you can find cargo ship data online for free), none of it is particularly valuable (you are looking for information, it is hard to know how much information is in cargo ship movement...it depends), and most non-quant hedge funds have been doing stuff like this for decades (i.e. hiring people to stand outside a retailer's stores and count customers)...most of this stuff is less useful than people think (again, you need information, data with intent).

Also, most of this stuff isn't in the price. Lots of people are collecting new data, it is definitely becoming more widespread but the actual synthesis is tricky (most people who are quants do not understand fundamentals, and most fundamental analysis don't understand data...most firms are swirling in a perfect storm of ignorance).

Theres actually still money to be made in small scale strategies. Sophisticated funds are running billions. They cant focus on strategies that only work for 100-500k. This is where big returns can be made. Even warren buffet will say, if he was only managing 1 million, he would get 100% a year returns.

That doesn't make sense unless there are very few viable small scale strategies, at which point they'd probably be difficult to identify. Your assertion might have been true before computers were able to help someone manage many strategies simultaneously.

No it makes a ton of sense. 100k is too small to have a researcher focus on full time. His compensation is probably 500k or more. Then add in fees, infra, cost of regulations, etc and small time strategies arent developed. Making money in the market is completely overrated from a difficulty perspective, the hard part is managing billions.

You're saying it'd require a salary of $500k or more to find someone competent enough to manage an investment opportunity worth $100k annually? That implies the investment opportunities are actually quite complicated. Obviously, opportunities exist for investment gains at all magnitudes. The question was whether they're easy enough for the average Jolanda. If Jolanda would require $500k annual salary to do the work, then, that easy opportunity doesn't really exist at all.

Alternately, if Jolanda doesn't need to make this opportunity her full-time job, then the investment company can pay her $500k to manage many of these small opportunities. After all, there must be many opportunities. Otherwise only 1 of the HN readers would be able to exploit it. In which case, again, it doesn't really exist at all.

Honestly, you seem like youre trying to reason this out with zero experience in the field. You cant just "run statistics to identify strategies". I'm not going to go into it further because you know nothing

I'm not sure how you came to either of those assumptions. First, I'm an economist. Second, I didn't suggest any method for identifying strategies. The question is the number of potential strategies.

If it's relatively easy to make money investing on a small scale, that means there are many potential strategies that don't require someone's full attention. If there are many potential strategies that don't require full attention, then someone skilled can go after many of those potential strategies and transform the small-scale into a large-scale opportunity. Since we've already established that these large-scale opportunities are hard ...

15% return on 5 billion is much higher than 100% return on 1 million. Its economics. Secondly, economics has nothing to do with investing, and is worthless. The sum of managing 1000 small strategies is way more work than the number you would manage with billions.

And yes, there is easy money laying around if you know what youre doing. There are dollar bills laying on the ground for smaller scale strats.

I don't see why those particular magnitude calculations are relevant to this analysis.

> economics has nothing to do with investing, and is worthless

Is it worthless because it has nothing to do with investing, or has nothing to do with investing because it's worthless? Further, why do you think the Federal Reserve hires so many economists if economics is worthless?

> And yes, there is easy money laying around if you know what youre doing. There are dollar bills laying on the ground for smaller scale strats. People just arent taking about them

Isn't that like saying, "Winning chess tournaments is easy if you know what you're doing."? If something requires expertise, then it isn't easy. If finding these smaller scale strategies requires expertise, then the expert probably has better things to do with their time. I suppose that holds to arguing with idiots on Hacker News.

I used to work on a team mixed of economics PhDs from Yale/Princeton/Harvard and with Phd Mathematicians. I know what economics is. All the best hedge funds are run by either math people or ruthless business men who cheat/activist invest/have non public information.

Also, the FED is in the process of destroying the US dollar and hyper inflating all the assets. Everyone but the capitalists are getting poorer.

The "Chief Economists" of many major US banks dont even hold economics degrees. People arent rational actors that you can plug into a math equation.

"If finding these smaller scale strategies requires expertise, then the expert probably has better things to do with their time. I suppose that holds to arguing with idiots on Hacker News."

Theres alot of friction of getting data to analyze, developing a strategy, executing the strategy, paying fees, managing risk, hedging. If you know how to do all that, you are probably already employed in that capacity or you run your own fund. But yes, if youre smart you can find an edge. Theres a ton of small man shops in NYC with 5 million in capital where everyone makes 600-1M a year.

> friction ... if youre smart you can find an edge

> easy money laying around

There's a bit of dissonance between the way you've characterized this process in different comments. That's my point.

"Hey, there's a dollar lying on the ground!"

"Can't be. Someone would have picked it up."

> destroying the US dollar and hyper inflating all the assets

Sounds like a good strategy to bring manufacturing back to the US. They've said the new policy is high employment.

"Hey, there's a dollar lying on the ground!"

Found the efficient market hypothesis believer, classic failing of economic theory.

Theres no alpha in the stock market!!!11!! If there was the completely efficient allocation of human capital, access of information, opportunity cost would strip it out so fast!!!

Yep, indeed I am. The same way I think casinos don't mind if you try to count cards at blackjack.

He said it pretty clearly. It's easy money if you look at it and work at it. The issue is most people don't have enough capital to try or don't have the balls to follow through. I've had many try my system and they can't hold the hard times.

Isn't that a contradiction in terms, "It's easy if you work at it."?

It's not easy to be Lebron James if you work at it. Also maybe it's just having a hidden talent you didn't know you had or a biological advantage.

no not really. you can use public information to give you an edge. And I say this as a person who trades and develops models at a vety successful market making firm.

Alternative data no one else can get easily certainly has tremendous value though.

Of course, predicting one or two seconds into the future (my primary concern) is easier than days or years, so there's that.

Granted, a typical Kaggle metagame-that-is-technically-against-the-rules is to use data from outside the dataset, which is one of the reasons winners have to be validated.

From: kaggle.com/c/jane-street-market-prediction/overview/code-requirements

"Freely & publicly available external data is allowed, including pre-trained models"

Generally this is allowed if you publish the data you're bringing in. They even create a sponsored thread for it in most competitions.

>Unless you have some kind of specialized non-public data (e.g satellite images of number of cars parked outside parking malls, number of cargo ships moving in and out)

Planet labs will sell you all of that data, in case people reading along here are curious.

I'm sure they've got more/better data in production. There seems to be some arbitrage that can be shaved off the edges for players with innovative enough strategies and good and timely enough data.

That’s not necessarily true.