Most Japanese kids end up going to cram schools to make up for what the school system (both public and private) lacks. The system sucks---long hours, Saturday classes are being reintroduced this spring, and the cram schools are usually at least 2 hours long---sometimes as much as 4 or 5 hours.
And Japan still can't get into the top 10 of the Education Index.
One aspect of the "family component" is also that the US is a much more unequal society than most western European countries, which is further exacerbated by the very local nature of US school funding. This gets lost in looking at averages.
Then, there is the family component. In a number of industrialized countries, education is highly prized and parents are deeply involved in their kids' education. Here, that is again not necessarily the case in many schools--and there is a limit to how much teachers can do to overcome that.
Also, I think that in many other industrialized countries, teachers are respected and paid reasonably compared to other professions. This draws talented people into the field (passion is important, but so is economics).