The problem is the divide between rich, middle class and poor. Poor kids just don't score nearly as high on tests than middle class and wealthy kids.
Of course, I don't have the stats or evidence, but I suspect if you took out our (USA) bottom 20% and the other countries bottom 20%, you would see a much more even comparison.
Just a theory.
While I know this is anecdotal, my wife is a reading/psychology teacher at a high school in Florida that is widely considered one of the worst, if not the worst, in the state. It's incredible the ways in which the students socioeconomic state plays into their ability and/or desire to learn. Everything from the students parents not being capable of or not willing to contribute to their educational growth to peer pressure from other students to not stand out (which includes appearing smarter than others, sadly).
My wife is lucky to be in a position where she can teach them in small groups of 1-3 students and get them away from that peer pressure. In these circumstances, and when someone is believing in their ability to do better, they really do excel and are incredibly bright. It's just like you said, their socioeconomic position has stacked the cards against them for a very long time and it becomes easy to fall into the same rut everyone else is in within those areas.
The U.S. does not have an education problem; it has a child poverty problem.