When applying CS/ML to finance, methodology is absolutely crucial. Yet most work - even work done by quants in major sell-side banks - pays little attention to methodology; it is often naively assumed that the methodology applied to classifying cat images can be directly transferred into the domain of finance.

Here is an article that explains the problem well: http://zacharydavid.com/2017/08/06/fitting-to-noise-or-nothi...

But that's because it is being applied by people who come from classifying cat images and have little domain knowledge.

>"When applying CS/ML to finance, methodology is absolutely crucial."

When wouldnt it be?

In other areas, we can often assume that each example in our dataset is an independent sample from some underlying distribution. We can also assume that the distribution stays constant. Furthermore, we have fairly high-accuracy ground truth labels. Also, we know that everything we need to classify a sample is in the data we are given. We may also have more data to play with than we really need.

Take handwritten digit recognition task like MNIST, for example - the way we write the number '5' does not change much, and the distribution of the different ways people write that number stays pretty much the same over time. The labels we are given are very accurate and everything we need to classify the image is in the data.

All these assumptions mean that picking the right methodology is fairly easy - anyone can read a short tutorial and get it right.

None of the assumptions I listed hold in the world of finance. There is no standard methodology that everyone can agree should be followed. You really have to be very, very careful in order to produce useful results.

It sounds like you just need to use a different methodology, not that methodology is unimportant for whatever ml 101 examples that became popular like MNIST, etc.

Yes, but I also think it's a lot easier to get it wrong in finance without even realizing it than in many other domains. Relatively more difficult to screw up the methodology on MNIST. Many mistakenly assume that the methodology from MNIST transfers directly to finance, which makes the problem worse.

ALl of that means that when talking about applying ML to finance, discussing the methodology in detail is a must. If it is not talked about or only briefly mentioned, from my experience that usually means that the methodology used is rubbish. Whereas you don't need to focus as much on the methodology when talking about MNIST-like problems - one can usually assume a reasonable one is used.

If you use financial performance to measure the success of a model, methodology is critical. It's easy to have models that generate outperformance on paper, in reality it turns out that nearly all of those had some flaw in them.

>"If you use financial performance to measure the success of a model, methodology is critical."

When wouldn't it be? And why does this sound exactly like the other post I just read?

If you compare to other results, small problems with timing usually don't impact results as severely. You don't get completely different results if you forget to move one time series by just a few hours.

As for the second part, I don't know. Probably because we had the same thought? Not sure which post you refer to.