I'm confused here. Both facts are salient (what was built and under which circumstances was it built) to any discussion of merit. The main complaint I see here is that admissions committees should use as much information as possible, which I doubt anyone disagrees with. What is racist is someone saying 'Oh, student 2 is black, thus without any further information, I'm going to assume he's poor and from Sudan'.

Case in point, we had a very wealthy black student in my college. This woman was not disadvantaged in any way, yet she played the race card all the time in order to claim a disadvantaged background. I'm talking about a family that would take their kids to France and England to summer. That level of wealth, yet framing all her accomplishments as if she came from the inner city. That's disingenuous, yet the (now-gone) affirmative action camp would have gladly taken her checking the 'African-American/Black' checkbox as a sign that all her accomplishments should be judged on a poor disadvantaged upbringing. How is that not racist?