So yes, you are allowed to discriminate on those sorts of things. If the attribute you are discriminating for is relevant to the job, it is ok to discriminate.
Take, for example, discrimination against the physically disabled - if a construction company requires that you are able to lift 50 pound bags of concrete, it wouldn't be guilt of unlawful discrimination because that is required for the job.
On the other hand, if you require someone be able to lift a 50 pound bag, but their job is to sit at a desk and program a computer... then you would be found guilty of unfair discrimination against the disabled.
IE age discrimination.
Either age discrimination is morally OK or it isn't.
I'd be fine with either the situation of age discrimination being disallowed or allowed, across the board, regardless of whether the discrimination is against young or old people.
Either way. I just want age discrimination against young and old people to be treated exactly the same.
Would you say the same thing if someone used similar dog whistles to talk about old people?
IE, "oh, being able to move quickly and think on your fit is important, and sometimes this can only be gained by being in good physical shape, and having good health, like a young person would have".
Either discrimination is OK or it isn't. One person's justifications aren't any different than another person's.