Why shouldn't those circumstances be taken into account? Perhaps it's because -- as a race -- most Asians are too proud to talk about this in their admissions essays, and moreover, their intellectual accumen would tend to indicate that these -- frankly minor -- issues are not important when considering what school to attend.
What we should really be talking about is why anyone -- Asian or black or whatever -- needs to elicit the sympathy of some old WASP to attend school. The whole 'write a sob story' trope is true because this appeals to popular white culture. It's time that ends, and we let adults run the show.
I also agree that the current situation of having to write a touching essay for some WASP is messed up. To be fair, not all essays are sob stories and people actually warn that it shouldn't be all sob and you need to show how you grew, but the whole idea of essays and personality is so vague and subjective, and kids get so stressed over writing a good essay.
EDIT: Offtopic, but looking at your username, do you happen to be the author of the Haskell BEAM library?
However, I think that this should stay as judging individuals and their circumstances and putting them in the context of the environment, and it should not justify wide-scale generalizations like secret quotas hidden with personality tests, etc. As you point out, such things further anti-intellectualism and penalize Asians and Jews for working hard.
(It would also be more productive to support people in those neighborhoods from the start, such as by supporting elementary schools, instead of waiting until college admissions.)