I spent an impressive amount of time with a biostats PhD (who had veto power on my IRB protocol) working this sort of stuff out. In the end it became clear he really didn't care about the machine's training process at all, and he was only interested in validation as a way to get to talk about comparitive statistics to evaluate the results of many machines with each other.

I drew many, many tables on the whiteboard that day.