The United States has been issuing passports by the millions for quite some time.
http://travel.state.gov/passport/services/stats/stats_890.ht...
Note that unlike some other countries, there is essentially no domestic use for a United States passport. United States citizens get passports if they are traveling overseas, and until recently travel to Canada and Mexico without passports was quite routine. So there are evidently plenty of Americans who have had a chance to go look around and learn from other countries. I would be quite happy to hear of more: in many respects, I think four years spent doing almost ANYTHING overseas and living in the local economy of a country with another language is more educational than four years spent pursuing a college degree, but meanwhile plenty of Americans, for business or for pleasure, do plenty of foreign travel.
Your point still holds, but too many people use that factoid as some sort of slight against Americans when it's really not.
In a country where most people have never even set foot in a modern european or asian city, the trouble is that most people don't have any idea how far behind US infrastructure has gotten.