> And then, the academic perspective is that prices should be modelled as random walks, though you may talk/learn about things such as "trend" and volatility.

The math involved in finance and economics always seems way behind that of other fields. The problem is that the other fields with more advanced math are so deep in theory that the people working in those areas are often either unaware of the potential real world applications of their work, or they are simply not interested in it (I’ve noticed there seems to be little overlap between the type of personality inclined to explore abstract theories as its own reward and the type of personality that prefers to apply existing knowledge to a real world problem).

> Suggesting that hidden variables/states/transitions can be learned from historical data is usually considered pseudo-scientific.

I mean, there’s a definitive answer to the question of stock market predictability. Unfortunately, it’s also uncomputable: if the conditional Kolmogorov complexity of a stock price time series given relevant auxiliary data is less than the size of the time series data (roughly speaking), then the stock price is predictable to some degree. Otherwise, it’s not.

I would be extremely skeptical if anyone claimed that stock price is truly Kolmogorov-random. However, I also think no single trading group’s algorithms (and data) are sufficiently more advanced than any other group’s to the point where algorithmic arbitrage is obvious to the market (or maintainable over a sufficiently long time period). I would not be surprised though if a sudden ML breakthrough destabilizes the entire market at some point in the near future when one group does in fact realize a step function improvement in their algorithms.

I think this is one of those cases where conflating "unpredictable" with "random" breaks down. As a matter of simplification, we treat many unpredictable processes as random if making the process at least somewhat predictable is sufficiently intractable. While this can change very quickly e.g. breaking encryption algorithms, for many data models the gains in making a process less unpredictable are more incremental and are largely dependent on having both better math and more efficient compute.

Unpredictability is as much a computational intractability frontier as it is a math problem. We know how to do approximately optimal prediction, but if you have to throw a supercomputer at the calculation and wait until the heat death of the universe to get an answer (which is the essential reality) then it has no value. But if you can grind out small improvements at the prediction frontier on a tractable amount of computing hardware due to algorithm advances and mathematical improvements in more narrow cases, then you have an almost unbounded greenfield to work with and these improvements will generalize well across diverse markets.

Predictability is defined in terms of probability distributions, and a price X is typically defined to be drawn from something like Xₜ₊₁ ~ Xₜ + N(μ,σ). The purpose might be to quantify some property of μ or σ or something like that. It means "X₀.ₜ does not hold any information about Xₜ₊₁". The assumption is that any price move reflects new information. It might be a reasonable model for various purposes.

But if this model was "true", RenTec would not work, and the "efficient market hypothesis" is invalidated. Which seems plainly obvious.

Ok, so if the market is not efficient, then it's actually a game (poker-like?) and zero sum. Academically, unthinkable thoughts.