It's like how red hair pops up in some families. The red hair gene never 'disappeared' or 'got averaged out', it just kept being passed on until it met another copy. Another example is interracial families: you might think that the kids would all 'look like' the average of the two parents, but sometimes one kid will look just like one parent even though they are biracial. (Apparently this can cause problems, like if a little blonde-haired blue-eyed girl is walking around with her dad from Africa.)

On a historical note, this was a major issue for Darwin & Galton trying to come up with a statistical model of 'genes': the initial efforts of 'blending inheritance' predicted that everyone would quickly become the same height etc, which is both obviously not true (short parents can have tall kids and vice versa) and would mean evolution is impossible (no variation to select on!).