Put another way, I'd take Einstein in his 40s over a thousand other randomly-chosen physicists in their 20s
(I don't think I've seen a more clear example of a false dichotomy.)
I agree that age is rarely the determining factor for someone's general competence: someone who's competent young is going to be competent when they're older. Their strengths do shift slightly, though.
Depending on the mission, experience may extremely valuable; when breaking new ground, sometimes creativity and energy are more important. It's hard to argue that there's no correlation with age there over a person's life, even if it's relatively minor.
Ageism to me seems a lot like sexism, racism, and every other -ism. The error is to use those characteristics as false metrics for someone's competence: e.g. "you're old, therefore you're slow" or "you're young, therefore you don't know what you're talking about". Let their work speak for itself and see if they're the right fit for the job at hand. The fact that there are age biases due to the type of work isn't always indicative of discrimination, though it often is.
(In Zuckerberg's case, I would be unsurprised to hear about discrimination given his YC07 talk about "hiring young").
If it's mainly selection based on metrics of certain types of competence and strengths that may correlate with age, then we shouldn't be hearing stories about people having to shave or dye their hair, get a hipster-wardrobe, just to have a chance at getting a second interview. (or plastic surgery ...)
Is this good or fair when it does happen? No.
Is it illegal? Sometimes, but making formal proof is daunting at best and usually not possible.
Thus, it comes down to this: treat others as you yourself would want to be treated. If a workplace reflects that ethic, it will be a place where old and young alike will want to work and where older workers will receive their due as befits their experience, skills, and performance, good or bad; if it does not reflect that ethic, it is a place to avoid if at all possible.
While Silicon Valley may represent the ageism problem in a particularly acute way owing to its youth culture, it really is no different from anywhere else. There is the good and the bad, and we each need to do what we can to uphold a high standard of common decency for ourselves and to make sure others do the same. It may not be a complete answer but it can help a lot in our immediate work environments and that will go a lot farther toward solving the problem than the law ever can.