What I don't understand about this theory, is, that you would then expect athletes of all sizes to die sooner... Certainly the heart of an athlete has gone a lifetime working harder than a non athlete... Maybe I'm misunderstanding something about the height/age correlation...

My guess, is your misunderstanding lies in biology in general. There is no constant maximum heart effort per lifetime. The heart is a muscle, if muscles are not used they become weak, conversely when muscles are used they become stronger and can grow (see: weight lifting). Also there is more to staying alive than a heart that beats.

Humans shrink in size as they grow older as well.

That may make sense, but appears to be mostly wrong.

The linked paper doesn't draw conclusions (it's just a survey paper), but does point to studies showing a positive correlation between cancer, rapid growth, and height.