There are plenty of females attracted to older males. The reverse is not nearly as true. There is a good genetic basis for this: young females that breed with an old male bring in whatever genetics cause long life. However old females are unable to breed at all so males that attempt to breed with older females fail and thus don't pass their genes on.

The above is really messy as it applies to the real world, but the basics are true enough.