The main thing I notice is that Koreans have slightly rounder faces, and smoother features. Koreans also tend to be slightly taller and more robust physically.

You are right though, the occasional Japanese person is going to be very hard to spot since they usually fall within the range of Korean diversity.