News for hackers. Stuff that matters.
Re:Goodbye, Data Science by chasely on Tuesday November 29, 2022 @09:57AM (#33788348)
I've been pitched by many "data-driven" vendors offering predictions. They often have very impressive accuracy metrics (RMSE, R2, etc). When I dive into the details these metrics are often reported using in-sample predictions.

I see this pointing to any of the following:

a) DS teams overpromising the accuracy of their approaches b) marketing driving the narrative and DS getting pulled along c) incompetence from the DS team

Re:Goodbye, Data Science by alexpetralia on Tuesday November 29, 2022 @12:51PM (#33790762)
Are these inferential statistics not designed to be in-sample?

I would imagine predictive statistics use more out-of-sample metrics like precision and recall.

Re:Goodbye, Data Science by antipaul on Tuesday November 29, 2022 @02:19PM (#33791893)
If you do it right…

That’s the problem: these metrics often come from overfitted or in-sample data, and are completely unrealistic when it comes to expected generalization performance.

I’m at the point where I never trust performance metrics anymore. Or rather, the worse they are, the more I trust them!

Re:Goodbye, Data Science by ZephyrBlu on Tuesday November 29, 2022 @08:42PM (#33795862)
I feel like you might be conflating a couple of things, though I'm not a DS so could be off base here.

My reading of the OP's description is that the vendors were offering interpolative predictions, but did not use a test/train split of data. This is in contrast to extrapolative predictions which I would call out-of-sample.

Thus due to not using a test/train split, they achieved extremely good accuracy because they were testing on the same data they trained on. Even though this is "in-sample", you can't use the same data for testing and training.