Calculating the sample size required for developing a clinical prediction model

Re:Calculating the sample size required for developing a cli... by bonoboTP on Monday September 07, 2020 @01:20PM (#24400897)

Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test examples into the training set and rejoyce that the new model can now deal with those previously difficult cases.

It's related to Goodheart's law: "When a measure becomes a target, it ceases to be a good measure".

It also works in one's personal life. If you find that you made a mistake, don't just fix that particular thing, go back and see what other similar things may need fixing. It connects to the idea of fixing the deeper reason than fixing the symptoms. "Deeper reason" just means, something that generalizes.

By the way, you need to be careful about your data split. This always depends on how you intend to use the model. If you intend to use it on new, unseen patients, then the train and test data cannot overlap with respect to people. Another obvious case is videos: You can't just take every even frame as a training sample and every odd frame as test. Even though technically you would be testing on non-overlapping data, the performance measure would be biased compared to real world performance. Or if you want to classify burglars from security camera footage, you may need to test it on new camera setups from different houses if you intend to deploy it to new houses. If your scenario is such that you'd perform training on each new site and run a location specific model, you can test on images from the same site.

You always have to use your brain to decide what to do.

[ Parent | on HN ]

Re:Calculating the sample size required for developing a cli... by shajznnckfke on Monday September 07, 2020 @02:43PM (#24401551)

> Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test examples into the training set and rejoyce that the new model can now deal with those previously difficult cases.

This is a great point, and it articulates some challenges that I’ve seen with test-driven development.

I would say that if you can write tests that fully specify the desired functionality, rather than merely check a few possible inputs, it’s less of an issue. This is a reason to try to build things with less knobs to twiddle, so the space of inputs has lower dimensionality.

[ Parent | on HN ]

Re:Calculating the sample size required for developing a cli... by virgilp on Monday September 07, 2020 @04:23PM (#24402470)

Yes but then the tests would formally describe the requirements - i.e.they would be a de-facto implementation. Congratulations, you just coded an (indirect) solution to your problem (a solution that you postulate is correct/bug-free)

[ Parent | on HN ]

Re:Calculating the sample size required for developing a cli... by tonyhb on Tuesday September 08, 2020 @09:42AM (#24408226)

Are you saying this like it's a bad thing?

Tests describing requirements acts as documentation, validation, and regression prevention with future refactors.

[ Parent | on HN ]

Re:Calculating the sample size required for developing a cli... by virgilp on Tuesday September 08, 2020 @10:01AM (#24408424)

My beef is with the assumption that tests can fully encode the requirements - to the point where any bug would trigger a test failure. Having such a test suite is no simpler than having a perfect implementation of the requirements - i.e. it's probably only feasible, at all, in the simplest/ "didactic" cases.

Having a perfect implementation is not bad, of course. It's just not realistic.

[ Parent | on HN ]

Re:Calculating the sample size required for developing a cli... by throwaway4007 on Monday September 07, 2020 @02:52PM (#24401641)

I was super into TDD and never thought that there was such a thing as "TDD going too far" until I read that post by an ex-Oracle employee: https://news.ycombinator.com/item?id=18442941

[ Parent | on HN ]

Re:Calculating the sample size required for developing a cli... by mumblemumble on Monday September 07, 2020 @05:40PM (#24403155)

This is a big reason why I prefer property testing to unit testing. One can think of focusing on the abstract behavior and invariants of the code, rather than just the expected outputs for specific inputs, as being somewhat akin to taking the time to plan out your experiment and the statistical tests you want to perform ahead of time, do your power analysis, etc. It's a fair bit more up-front work, but helps to ensure that you have a crystal clear idea of what you're going to do before you start doing it.

[ Parent | on HN ]