I wonder if that explains why women are on average longer lived than men - is it just the average height difference?

I don't know. AFAIK, short men have a higher rate of heart disease than tall men. http://time.com/3815022/shorter-people-heart-disease/

Makes sense. The heart wouldn't need to be as strong to circulate blood, and by extension not need to be as large or need as much nourishment either.

What I don't understand about this theory, is, that you would then expect athletes of all sizes to die sooner... Certainly the heart of an athlete has gone a lifetime working harder than a non athlete... Maybe I'm misunderstanding something about the height/age correlation...

My guess, is your misunderstanding lies in biology in general. There is no constant maximum heart effort per lifetime. The heart is a muscle, if muscles are not used they become weak, conversely when muscles are used they become stronger and can grow (see: weight lifting). Also there is more to staying alive than a heart that beats.

Humans shrink in size as they grow older as well.

That may make sense, but appears to be mostly wrong.

The linked paper doesn't draw conclusions (it's just a survey paper), but does point to studies showing a positive correlation between cancer, rapid growth, and height.