There's a network effect at Caltech, which does not practice AA, because it's so small. Basically, if you get into Caltech without AA, chances are you got into an equally good university with AA.

However, it's clear to the outside observer that your single-digit hispanic or black prospective classmates at Caltech are generally more miserable than your prospective classmates at MIT, Stanford, Harvard, etcetera. They are usually the only person like you in their dorm, or in their year, or in their major. So you don't go to Caltech.

In the 70s admissions actively canvassed South LA high schools.