Yup, this thread has a nice overview of ways performance on a validation set can overestimate clinical performance:

https://twitter.com/IAmSamFin/status/1122271463170564100

Another example of change over time:

> One difficulty in such a comparison is that Gleason grading standards have shifted over time, so that scores below six are now rarely assigned, and assigning a higher grade has become more common

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3775342/