I'd guess class size and teacher competence are the two main institutional factors (which separate expensive private schools with ~15 kids per teacher from public schools with ~30), and enough food, sleep, parental involvement, extracurricular activites etc. with respect to the home living situation are the major factors. Individual attention makes a big difference when it comes to lifting average performance, as students falling behind can get help before it reaches the drop-out level.

On the 'cultural expectations' front, rural white America has some similar issues to urban black America, of the 'why do you want to go to college' or 'being good at math is for nerds' or 'take your nose out of the books and play more sports' variety.

My own rather jaded cynical view is that this is a setup deliberately engineered by the ruling class, as if everyone could do math well nobody would ever have bought subprime-adjustable-rate mortgage loans, as they'd have realized they'd be better off just saving money by renting until they could afford a fixed-rate loan. The ability to estimate probabilities and to calculate compound interest should be required skills for high-school graduates.

An ignorant population is more easily fleeced - and let's not forget why it was a crime to teach slaves to read and write in the Old South, either. The recent attempts to lower math standards for high school students in California are probably motivated by that kind of agenda.