Re:Test machine learning the right way: regression testing
by saladtoes on Tuesday July 06, 2021 @08:53AM
(#27747836)
“The latter being when the training or test data follows a different distribution to the in-operation data”
This form of ML bug is the most challenging to catch. The true in-operation distribution is often unknown which makes testing for such bugs a very challenging problem. Any thoughts on this?
Re:Test machine learning the right way: regression testing
by mklond on Tuesday July 06, 2021 @09:00AM
(#27747919)
Thanks for your comment. The whole field of run-time monitoring is concerned with this problem. It's a tough one to crack when the distribution changes are subtle, but you can and should at least check simple data attributes for consistency.
This form of ML bug is the most challenging to catch. The true in-operation distribution is often unknown which makes testing for such bugs a very challenging problem. Any thoughts on this?