IMHO, those are often rationalizations - the real reason is often that an older worker must have obviously 'failed' in some way to be asking for the job in the first place.

It's threatening to people's narrative that if they work hard, they will be rewarded. So, obviously, an older worker must not be a diligent worker.