It is racist if the meaning is "we'd be scoring fine if it wasn't for them dragging us down with their genetic inferiority about which nothing can be done" rather than taken as a reason for improving schooling for the poor and otherwise disadvantaged.