The most interesting aspect of this is backtesting. Quant models get run on past data to see if their predictions work.

When you use LLMs agents, though, you run into their memorized knowledge of the world. And then there's the fact that they do their research on the open internet. It makes backtesting hard - but not impossible.

We wrote about how we do our pastcasting validation here: https://stockfisher.app/backtesting-forecasts-that-use-llms